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Abstract 


The  Innovative  Naval  COMbat  MANagement  Decision  Support  (INCOMMANDS)  Technology 
Demonstration  Project  (TDP)  attempts  to  improve  the  performance  of  Threat  Evaluation  (TE)  and 
Combat  Power  Management  (CPM)  functions  in  response  to  multiple  threats  and  impediments 
introduced  by  the  littoral  environment.  Specifically,  the  purpose  of  the  INCOMMANDS  TDP  is 
to  develop  and  demonstrate  advanced  Above  Water  Warfare  (AWW)  command  decision  support 
concepts  for  the  command  team  of  the  Halifax  Class  Frigate  in  order  to  improve  the  overall  TE 
and  CPM  decision-making  effectiveness.  This  report  presents  preliminary  validation  results 
stemming  from  a  Heuristic  Evaluation  by  a  Human  Factors  analyst  and  Usability  and  Utility 
Testing  with  Naval  operators  of  the  INCOMMANDS  Command  Decision  Support  Capability 
(CDSC)  prototype.  The  results  of  the  usability  and  utility  evaluation  suggest  that  the  concepts 
presented  in  the  INCOMMANDS  CDSC  would  improve  task  performance,  increase  situation 
awareness,  and  decrease  operator  workload.  Further  analysis  and  evaluation  efforts  are  required 
to  substantiate  this  finding. 


Resume 


Ee  projet  de  demonstration  technologique  (PDT)  INCOMMANDS  (Innovative  Naval  COMbat 
MANagement  Decision  Support,  en  anglais)  vise  a  ameliorer  refficacite  des  fonctions 
d’ evaluation  des  menaces  et  de  gestion  de  la  puissance  de  combat  dans  le  cadre  de  scenarios  de 
menaces  multiples  et  d’obstacles  introduits  par  le  milieu  littoral.  Plus  precisement,  le  PDT 
INCOMMANDS  vise  a  elaborer  et  a  demontrer  des  concepts  avances  d’aide  aux  prises  de 
decisions  de  commandement  en  situation  de  guerre  aerienne  et  de  surface  pour  Tequipe  de 
commandement  des  fregates  de  classe  Halifax  afm  d’ ameliorer  ses  capacites  d’ evaluation  de  la 
menace  et  de  gestion  de  la  puissance  de  combat.  Ee  present  rapport  expose  les  resultats  de 
validation  preliminaire  d’une  evaluation  heuristique  par  un  analyste  des  facteurs  humains  et  d’une 
serie  d’essais  de  convivialite  et  de  fonctionnalite  effectues  avec  des  operateurs  navals  du 
prototype  d’aide  a  la  decision  de  commandement  developpe  dans  le  cadre  du  projet 
INCOMMANDS.  Ees  resultats  de  cette  evaluation  suggerent  que  les  concepts  mis  de  Tavant  par 
le  PDT  INCOMMANDS  permettraient  d’ameliorer  I’accomplissement  des  taches,  d’augmenter 
la  connaissance  de  la  situation  et  de  reduire  la  charge  de  travail  des  operateurs.  D’autres  analyses 
et  evaluations  seront  necessaires  pour  confirmer  ees  constatations. 
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Executive  summary 


INCOMMANDS  TDP:  Human  Factors  Evaluation  of  the  Command 
Decision  Support  Capability  Prototype 

Kevin  Baker  and  Lisa  Hagen;  DRDC  Toronto  CR  2009-041;  Defence  R&D 
Canada  -  Toronto;  March  2009. 

Introduction  or  background:  The  Innovative  Naval  COMbat  MANagement  Decision  Support 
(INCOMMANDS)  Technology  Demonstration  Project  (TDP)  attempts  to  improve  the 
performance  of  Threat  Evaluation  (TE)  and  Combat  Power  Management  (CPM)  functions  in 
response  to  multiple  threats  and  impediments  introduced  by  the  littoral  environment. 
Specifically,  the  purpose  of  the  INCOMMANDS  TDP  is  to  develop  and  demonstrate  advanced 
Above  Water  Warfare  (AWW)  command  decision  support  concepts  for  the  command  team  of  the 
Halifax  Class  Frigate  in  order  to  improve  the  overall  decision-making  effectiveness  for  TE  and 
CPM 

This  report  describes  results  stemming  from: 

1.  A  Heuristic  Evaluation  verifying  compliance  of  the  INCOMMANDS  Command  Decision 
Support  Capability  (CDSC)  design  with  guidelines  and  principles  stipulated  in  the 
INCOMMANDS  TDP  Human  Factors  Design  and  Evaluation  Guide. 

2.  Usability  and  Utility  Testing  of  the  INCOMMANDS  CDSC  prototype  with  Naval  operators 
in  order  to  rate  the  usability  of  the  system,  their  comfort  level  when  using  the  system,  their 
overall  workload  and  situation  awareness  when  using  the  system,  and  their  overall 
satisfaction  with  the  system. 

Results  &  Significance:  Based  on  the  evaluation  of  the  INCOMMANDS  CDSC,  it  was 
concluded  that: 

1.  Both  evaluation  techniques  provide  complementary  insight  into  improving  the 
INCOMMANDS  CDSC.  The  Heuristic  Evaluation  helps  to  ensure  that  the  design  of  system 
is  compliant  with  HCl  best  practices.  Whereas  the  Usability  and  Utility  Testing  with  Naval 
operators  ensures  that  contextual  feedback  is  obtained  which  is  lacking  from  the  former 
evaluation  technique. 

2.  The  results  from  the  Heuristic  Evaluation  provide  specific  direction  to  support  future 
development  of  the  INCOMMANDS  CDSC  in  order  to  improve  the  overall  usability  of  the 
system.  As  such,  any  additional  work  to  be  completed  on  the  system  should  attempt  to 
improve  the  usability  of  the  system  by  both  fixing  the  identified  non-compliances  as  well  as 
adding  the  missing  functionality. 

3.  Based  on  the  participant  feedback  received  during  the  Usability  and  Utility  Testing,  the 
following  preliminary  conclusions  can  be  postulated  (additional  studies  are  recommended  to 
further  validate  these  claims): 
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a.  The  proposed  INCOMMANDS  CDSC  OMl  concepts  are  in  line  with  the  current 
operational  requirements  of  the  Operations  Room  Officer  (ORO)  and  Sensor 
Weapons  Controller  (SWC)  Naval  operators  and  would  make  their  job  easier. 

b.  The  participants  positively  rated  ease  of  use  of  the  system,  their  comfort  level  when 
using  the  system,  their  understanding  of  what  the  system  was  doing  at  all  times,  and 
overall  satisfaction  with  the  system. 

c.  Overall  workload  would  decrease  when  using  the  INCOMMANDS  CDSC  compared 
to  using  only  the  physical  display  for  TE  and  CPM  activities.  Furthermore,  overall 
workload  would  decrease  in  a  high  workload  situation  relative  to  a  low  workload 
situation. 

d.  Overall  situation  awareness  would  improve  when  using  the  INCOMMANDS  CDSC 
compared  to  using  only  the  physical  display.  Overall  situation  awareness  would 
increase  in  a  high  workload  situation  compared  to  a  low  workload  situation. 

e.  The  INCOMMANDS  CDSC  can  be  employed  in  situations  subject  to  time  pressure. 

f  The  INCOMMANDS  CDSC  would  not  interfere  with  the  flow  of  communications  in 
the  operations  room. 

Future  plans:  Further  analysis  and  evaluation  efforts  are  required  to  substantiate  the  findings 
from  the  usability  and  utility  trial.  Future  evaluations  should  include  the  collection  of  objective 
performance  measures  using  the  experimental  plan  outlined  in  Annex  F.  As  well,  a  follow-up 
heuristic  evaluation  should  be  conducted  if  the  CDSC  is  frilly  implemented  to  confirm 
compliance  with  the  INCOMMANDS  TDP  Human  Factors  Design  and  Evaluation  Guide. 
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INCOMMANDS  TDP:  Human  Factors  Evaluation  of  the  Command 
Decision  Support  Capability  Prototype 

Kevin  Baker  and  Lisa  Hagen;  DRDC  Toronto  CR  2009-041;  Defence  R&D 
Canada  -  Toronto;  March  2009. 

Introduction  on  contexte:  Le  projet  de  demonstration  technologique  (PDT)  INCOMMANDS 
(Innovative  Naval  COMbat  MANagement  Decision  Support,  en  anglais)  vise  a  ameliorer 
refficacite  des  fonctions  d’evaluation  des  menaces  et  de  gestion  de  la  puissance  de  combat  dans 
le  cadre  de  scenarios  de  menaces  multiples  et  d’ obstacles  introduits  par  le  milieu  littoral.  Plus 
precisement,  le  PDT  INCOMMANDS  vise  a  elaborer  et  a  demontrer  des  concepts  avances  d’aide 
aux  prises  de  decisions  de  commandement  en  situation  de  guerre  aerienne  et  de  surface  pour 
Tequipe  de  commandement  des  ffegates  de  classe  Halifax  afin  d’ameliorer  ses  capacites 
d’evaluation  de  la  menace  et  de  gestion  de  la  puissance  de  combat. 

Le  present  rapport  decrit  les  resultats  des  projets  suivants  : 

1.  Evaluation  heuristique  visant  a  verifier  le  degre  de  conformite  de  la  conception  de  capacite 
d’aide  aux  decisions  de  commandement  avec  les  lignes  directrices  et  les  principes  enonces 
dans  le  Guide  de  conception  et  d’evaluation  des  facteurs  humains  du  PDT  INCOMMANDS. 

2.  Essais  de  convivialite  et  de  fonctionnalite  du  prototype  CDSC  INCOMMANDS  entre  les 
mains  d’operateurs  navals  dans  le  but  d’evaluer  la  facilite  d’utilisation  perdue,  le  degre  de 
confort,  la  charge  de  travail  globale  et  la  connaissance  de  la  situation  par  les  operateurs,  ainsi 
que  leur  degre  de  satisfaction  generate  a  I’egard  de  ce  systeme. 

Resultats  et  consequences:  II  ressort  de  cette  evaluation  du  prototype  CDSC  INCOMMANDS 
que: 

1.  Ees  deux  techniques  d’evaluation  foumissent  des  perspectives  complementaires 
d’ amelioration  de  la  CDSC  INCOMMANDS.  E’evaluation  heuristique  permet  de  s’assurer 
que  la  conception  du  systeme  est  conforme  aux  pratiques  recommandees  en  matiere 
d’interfaces  homme -machine.  Pour  ce  qui  est  des  essais  de  convivialite  et  de  fonctionnalite 
avec  des  operateurs  navals,  ils  foumissent  les  reactions  humaines  en  contexte  qui  manquent 
dans  E autre  technique  d’evaluation. 

2.  Ees  resultats  de  revaluation  heuristique  foumissent  des  indications  specifiques  pour 
r orientation  des  futurs  developpements  de  la  CADC  INCOMMANDS  en  vue  d’ameliorer  la 
convivialite  du  systeme.  Dans  cet  esprit,  tous  les  travaux  futurs  a  effectuer  sur  le  systeme 
devront  viser  a  ameliorer  sa  convivialite  en  corrigeant  les  non-conformites  identifiees,  en  plus 
d’incorporer  les  fonctionnalites  manquantes. 

3.  Ees  commentaires  des  participants  aux  essais  permettent  de  tirer  les  conclusions  preliminaires 
suivantes  (qui  devraient  cependant  etre  validees  par  des  etudes  complementaires)  : 
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a.  Les  concepts  d’interface  operateur-systeme  proposes  pour  la  CDSC  INCOMMANDS 
conviennent  aux  exigences  operationnelles  actuelles  des  operateurs  navals  OSO  et 
SWC  et  vont  dans  le  sens  de  I’aHegement  de  leur  charge  de  travail. 

b.  Les  participants  ont  juge  favorablement  la  facilite  d’utilisation,  se  sont  sends  a  I’aise 
lorsqu’ils  ont  travaille  avec  le  systeme,  ont  eu  un  bon  niveau  de  comprehension  de 
son  fonctionnement  et  se  sont  declare  globalement  satisfaits. 

c.  L’utilisation  de  la  CDSC  INCOMMANDS  permettra  de  reduire  la  charge  de  travail 
globale  par  rapport  a  rutilisation  des  affichages  physiques  seuls  pour  les  activites  de 
TE  et  de  CPM.  De  plus,  le  systeme  devrait  permettre  de  reduire  la  charge  de  travail 
dans  une  situation  a  haute  intensite  comparativement  a  une  situation  de  faible 
intensite. 

d.  L’utilisation  de  la  CDSC  INCOMMANDS  permettra  d’assurer  un  meilleur  eveil 
situationnel,  toujours  par  rapport  aux  affichages  physiques  seuls.  L’ eveil  situationnel 
global  serait  ameliore  dans  une  situation  a  haute  intensite  comparativement  a  une 
situation  de  faible  intensite. 

e.  La  CDSC  INCOMMANDS  pent  etre  employee  dans  des  situations  impliquant  des 
contraintes  de  temps  critiques. 

f.  La  CADC  INCOMMANDS  n’a  pas  d’incidence  negative  sur  les  communications  qui 
se  deroulent  dans  la  salle  des  operations. 

Plans  pour  I’avenir:  11  faudra  prevoir  d’autres  efforts  d’analyse  et  d’evaluation  pour  confirmer 
les  enseignements  tires  des  essais  de  convivialite  et  de  fonctionnalite.  Ces  evaluations  futures 
devraient  prevoir  la  collecte  de  mesures  de  performance  objectives,  comme  prevu  dans  le  plan 
experimental  expose  dans  Annex  F.  Enfin,  si  la  CDSC  est  completement  implantee,  il  faudra 
prevoir  une  evaluation  heuristique  afm  de  confirmer  le  degre  de  conformite  avec  le  Guide  de 
conception  et  d’evaluation  des  facteurs  humains  du  PDT  INCOMMANDS. 
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1  Introduction 


1.1  Background 

Operations  by  the  HALIFAX  Class  frigates  expose  the  ship  to  an  array  of  risks  that  require  the 
identifieation  and  prioritization  of  threats  and,  if  required,  the  application  of  combat  power 
resources  to  counter  the  threat’s  intent  to  inflict  harm.  Currently,  these  functions  are  performed 
by  a  select  number  of  individual  in  the  operations  room  via  a  series  of  cognitive  processes.  While 
the  operators  are  reasonably  adept  at  performing  these  functions  for  a  single  threat,  their  ability  to 
effectively  achieve  similar  results  for  a  multi-threat,  multi-axis  threat  is  severely  hampered. 
Furthermore,  the  HALIFAX  Class  ships  are  increasingly  required  to  conduct  operations  in  a 
littoral  environment,  an  operational  environment  characterized  by  high  density  small  craft  traffic, 
land-based  threats,  impeding  meteorological  conditions,  and  difficult  terrain  for  naval  operations. 
This  cluttered  littoral  environment  provides  an  operational  environment  that  is  conducive  to 
attacks  from  asymmetric  threats  such  as  suicide  attack  by  a  small  boat  or  swarms  of  small  boats 
carrying  heavy  or  small  arms,  low  and  slow  flyers,  and  a  wide  range  of  underwater  mines.  In  turn, 
this  provides  a  new  set  of  challenges  for  the  Canadian  Navy  as  the  time  and  space  for  a  ship  to 
detect  and  react  to  the  threats  is  effectively  reduced. 

The  Innovative  Naval  COMbat  MANagement  Decision  Support  (INCOMMANDS)  Technology 
Demonstration  Project  (TDP)  attempts  to  improve  the  performance  of  Threat  Evaluation  (TE)  and 
Combat  Power  Management  (CPM)  functions  in  response  to  multiple  threats  and  impediments 
introduced  by  the  littoral  environment.  Specifically,  the  purpose  of  the  INCOMMANDS  TDP  is 
to  develop  and  demonstrate  advanced  Above  Water  Warfare  (AWW)  command  decision  support 
concepts  for  the  command  team  of  the  Halifax  Class  Frigate  in  order  to  improve  the  overall  TE 
and  CPM  decision-making  effectiveness. 

Specific  objectives  of  the  TDP  are  to: 

1.  develop  and  demonstrate  advanced  AWW  command  decision  support  concepts  in  a  manner 
that  will  assist  the  Halifax  Class  Modernization  (HCM)/FEEEX  project  define  specifications 
for  TE  and  CPM  functions  that  are  practicable  for  Canadian  industry, 

2.  elicit  the  Canadian  Navy’s  cognitive/decision  support  and  information  requirements  to 
perform  single  ship  AWW  command  and  control, 

3.  develop  a  flexible  and  robust  software  architecture  that  enables  the  integration  of 
heterogeneous  algorithms  and  incremental  enhancements, 

4.  develop  a  knowledge -based  framework  that  allows  the  efficient  exploitation  of  a  priori 
information  and  improves  both  human  and  automated  TE  and  CPM  functions, 

5.  develop  comprehensive  evaluation  methods  and  metrics  (measures  of  performance  (MOPs) 
and  measures  of  effectiveness  (MOEs))  that  permit  the  empirical  validation  and  assessment 
of  new  decision  support  systems  and  human  decision-making  effectiveness, 

6.  develop  an  advanced  naval  command  and  control  (C2)  modeling  and  simulation  capability 
that  will  be  compatible  with  and  of  interest  to  the  Canadian  Forces  Maritime  Warfare  Centre 
(CFMWC),  and 
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7.  explore  multi-ship  CPM  concepts  in  order  to  support  the  Canadian  Navy’s  contribution  to  the 
international  Battle  Management  Command,  Control,  Communications,  Computers,  and 
Intelligence  (BMC41)  project  through  a  Task  Group  conceptual  study. 

1.2  Objective 

In  support  of  objective  2  of  the  INCOMMANDS  TDP,  CAE  PS  elicited  information  from 
Canadian  Navy  Subject  Matter  Experts  (SMEs)  regarding  the  conduct  of  C2  functions  on-board 
the  HAEIFAX  Class  frigate.  Results  of  this  effort  are  presented  in  the  INCOMMANDS  System 
Analysis  Report  [5].  Further  to  this  initial  system  analysis  effort,  a  concept  of  operation  and 
preliminary  OMl  design  concepts  for  the  INCOMMANDS  Command  Decision  Support  Concept 
(CDSC)  were  developed  in  accordance  with  objective  1  of  the  INCOMMANDS  TDP.  These 
results  are  presented  in  the  INCOMMANDS  TDP:  OMl  Design  Concepts  report  [3]. 

As  a  follow  on  to  these  efforts,  the  following  report  documents  the  findings  stemming  from  an 
evaluation  of  the  INCOMMANDS  CDSC  OMl.  The  purpose  of  the  present  investigation  was  to: 

1.  Perform  a  heuristic  evaluation  of  the  INCOMMANDS  CDSC  to  insure  that  specific  elements 
of  its  design  are  compliant  with  the  INCOMMANDS  TDP:  Human  Factors  Design  and 
Evaluation  Guide  [2],  and 

2.  Collect  utility  and  usability  data  to  be  used  in  the  development,  validation  and 
implementation  of  decision  support  concepts  to  support  the  conduct  of  threat  evaluation  and 
application  of  combat  power  functions  by  Canadian  Forces  (CF)  personnel  on-board  a 
HAEIFAX  Class  frigate. 

Originally,  the  purpose  of  the  study  was  to  conduct  a  laboratory-based  experiment  requiring 
participants  to  perform  activities  and  make  decisions  to  support  goals  pertaining  to  threat 
evaluation  and  management  of  combat  power.  Situational  awareness,  workload,  task  performance 
and  usability  data  were  to  be  collected  during  the  experiment  (Annex  F);  however,  the  current 
INCOMMANDS  prototype  was  not  able  to  support  the  experimental  demands  due  in  part  to  time 
budget,  and  SME  availability  constraints.  Therefore,  the  evaluation  was  limited  to  Usability  and  a 
Utility  Testing  to  assess  the  system’s  functionality,  judge  the  effect  of  the  interface  on  the  user, 
and  identify  usability  and  utility  problems. 

1.3  This  Document 

This  document  describes  the  usability  and  utility  trial  methodology,  the  plan  for  conducting  the 
Human  Factors  (HF)  evaluation  of  the  INCOMMANDS  CDSC,  the  results  of  these  subjective 
evaluations,  and  recommendations  for  future  work.  The  report  consists  of  the  following  sections: 

1.  Section  One  provides  background  information  and  the  objectives  of  the  trial; 

2.  Section  Two  provides  background  details  regarding  Naval  C2  functions  and  INCOMMANDS 
CDSC; 

3.  Section  Three  outlines  the  evaluation  methodology  utilized  for  the  INCOMMANDS  TDP  as 
well  as  the  process  for  both  the  Heuristic  Evaluation  and  Usability  and  Utility  Testing; 

4.  Section  Four  summarized  the  results  stemming  from  the  Heuristic  Evaluation; 
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5.  Section  Five  presents  the  results  of  the  Usability  and  Utility  Testing  of  the  INCOMMANDS 
CDSC; 

6.  Section  Six  details  overall  conclusions  in  regards  to  the  conduct  of  the  overall  evaluation  of 
the  system. 

7.  Annex  A  provides  the  compliance  matrix  that  captures  the  results  from  the  Heuristic 
Evaluation  of  the  INCOMMANDS  CDSC; 

8.  Annex  B  presents  the  participant  Informed  Consent  Form  for  the  Usability  and  Utility 
Testing; 

9.  Annex  C  provides  the  Threat  Evaluation  Usability  Questionnaire  for  the  Usability  and  Utility 
Testing; 

10.  Annex  D  presents  the  Combat  Power  Management  Usability  Questionnaire  for  the  Usability 
and  Utility  Testing; 

11.  Annex  E  presents  the  Overall  Utility  Questionnaire  for  the  Usability  and  Utility  Testing;  and 

12.  Annex  F  provides  a  future  experimental  plan  including  informed  consent  and  experimental 
questionnaires. 
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2  Background 


2.1  General 

As  background  to  the  conduct  of  both  the  Heuristic  Evaluation  and  Usability  and  Utility  Testing, 
the  following  sections  provide  a  high  level  overview  of  the  concepts  surrounding  Naval  C2 
functions  and  the  INCOMMANDS  CDSC. 


2.2  Naval  C2  Functions 

The  fundamental  objective  of  the  project  is  focus  on  the  conduct  of  TE  and  CPM  functions, 
primarily  by  the  Operations  Room  Officer  (ORO)  and  Sensor  Weapons  Controller  (SWC) 
onboard  the  HAEIFAX  Class  frigate,  as  it  pertains  to  AWW.  As  depicted  in  Figure  1,  the  naval 
C2  process  can  be  decomposed  into  a  set  of  generally  accepted  functions  that  must  be  executed 
within  some  reasonable  delays  to  ensure  mission  success  [10].  These  functions  include:  Picture 
Compilation;  Threat  Evaluation;  Engageability  Assessment;  and  Combat  Power  Management. 


Sensors 

Deterrence 

Navigation 

Wcap<ins 

Figure  1:  Naval  C2  Process  [10] 


While  the  picture  compilation  function  is  an  essential  prerequisite  for  supporting  the  performance 
of  the  remaining  three  functions,  it  was  not  the  focus  of  the  INCOMMANDS  TDP.  It  has  been 
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included  in  this  section  for  completeness.  A  high-level  description  of  those  functions,  related  to 
battlespace  management,  is  given  in  the  following  subsections.  The  intent  is  to  introduce  these 
concepts  while  focusing  on  the  conduct  of  these  functions  within  the  context  of  a  single  platform 
and  multi-platform  scenarios  in  subsequent  sections. 

2.3  Picture  Compilation 

In  all  maritime  operations,  ranging  from  peacetime  through  wartime,  a  fundamental  requirement 
is  the  compilation  of  a  plot  of  surface,  air,  and  subsurface  tracks.  The  process  of  all  actions  and 
activities  aimed  at  compiling  a  plot  is  referred  to  as  picture  compilation.  In  maritime  operations, 
picture  compilation  will  normally  be  executed  to  support  decision  making  in  relation  to  the 
mission.  The  nature  of  the  mission  will  dictate  the  importance  of  the  plot  and  what  information  is 
to  be  derived  from  it. 

Picture  compilation  involves  the  following  sub-processes  [11]  as  represented  in  Figure  2  [10]: 

1.  Object  Detection  includes  the  employment  of  sensors  in  a  volume  of  interest  in  order  to 
determine  the  presence  or  absence  of  objects  or  object-related  data; 

2.  Localization  (or  Object  Tracking)  includes  the  employment  of  sensors  to  determine  the 
positional  information  and  movements  of  an  object;  and 

3.  Object  Recognition  includes  the  employment  of  sensors  to  determine  characteristics  of  an 
object.  Comparing  the  collected  characteristics  against  reference  (or  a  priori)  data  can  lead  to 
correlation  with  a  level  of  confidence.  Object  Identification  includes  the  assignment  of  one 
of  the  six  standard  identities  to  a  detected  contact  (hostile,  suspect,  unknown,  neutral, 
assumed  friend,  friend). 

Sensors  available  for  picture  compilation  include  those  organic  to  a  particular  platform  (e.g., 
radar,  electro-optical  (EO),  electronic  support  measures  (ESM),  acoustics)  as  well  as  assets  that 
are  assigned  as  direct  support  (e.g.,  submarines,  marine  patrol  aircraft).  Data  links,  voice 
communications,  and  messages  are  used  to  transfer  object  information  between  two  or  more 
platforms  during  task  group  or  coalition  operations. 

2.4  Threat  Evaluation 

Threat  evaluation  establishes  the  current  intent  and  capabilities  of  non-friendly  entities  within  the 
volume  of  interest  based  on  a  priori  information,  the  tactical  picture  (and  track  database), 
available  intelligence,  constraints,  and  data  received  from  complementary  sources  in  relation  to 
the  mission  objectives  (Figure  3).  It  is  an  ongoing  process  of  determining  if  an  entity  intends 
(i.e.,  threat  intent)  and  is  able  to  (i.e.,  threat  capability)  inflict  evil,  injury,  or  damage  to  the 
defending  forces  and/or  their  interests,  along  with  the  prioritized  ranking  of  such  entities 
according  to  the  level  of  threat  they  pose  to  ownship  [1].  This  output  is  intended  to  support  the 
determination  of  the  ability  of  the  ownship  to  engage  a  specific  threat. 
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Figure  2:  Picture  Compilation  [10] 


Threat  Intent  Assessment 


Q-^  Threat  Evaluation 


Threat  Capability 
Assessment 


Figure  3:  Threat  Evaluation  [10] 


Collated  information  from  all  available  sources  is  interpreted  as  part  of  the  overall  analysis  of 
threat  information  in  an  attempt  to  discern  patterns  which  may  provide  clues  as  to  the: 
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1 .  Intent  of  the  threat.  This  is  a  determination  of  whether  the  threat  is  displaying  hostile  intent 
(or  not)  with  respect  to  a  target.  Hostile  intent  translates  into  the  will  or  determination  of  a 
threat  to  inflict  harm  or  injury  [1]. 

2.  Capability  of  the  threat.  This  is  a  determination  of  the  capability  of  a  live  track  and 
involves  evaluating  both  its  opportunity  and  lethality.  Opportunity  involves  evaluating  if  and 
when  the  threat  has  the  ability  to  attack  its  target.  Lethality  is  static  information  regarding  the 
level  of  damage  a  threat  can  deliver  if  its  combat  power  is  applied  against  a  target.  The  threat 
will  only  have  the  opportunity  to  deliver  its  lethality  provided  the  following  conditions  are 
satisfied:  the  threat  has  sufficient  energy  to  reach  the  target,  the  threat  can  detect  and  track  the 
target,  and  physical  obstructions  do  not  impede  access  to  the  target  [1]. 

A  suite  of  radar,  ESM,  and  acoustic  analysis  and  interpretation  tools  may  also  be  at  the  operator’s 
disposal  to  assist  in  the  performance  of  this  task. 

2.5  Engageability  Assessment 

Within  the  volume  of  interest,  operators  will  evaluate  the  feasibility  of  own  force’s  engagement 
options  against  the  non-friendly  entities  (Figure  4).  This  process  is  intended  to  assist  the 
application  of  combat  power  by  eliminating  candidate  solutions  that  violate  one  or  more  hard 
constraints  and  therefore  are  not  feasible  for  execution  [10]. 

This  process  takes  into  account  a  series  of  inputs  of  the  following  nature  [10]: 

1.  Mission  Restraints,  Impacts  mission  constraints  on  combat  power  (CP)  deployment 
including  rules  of  engagement  (ROE);  mission  objectives;  other  warfare  in  progress  or 
planned;  and  tactical  doctrine  based  on  a  priori  mission  planning;  and 

2.  Own  Capability  Assessment,  This  involves  the  estimation  of  the  performance  of  combat 
power  resources  against  individual  threats.  This  requires  the  evaluation  of  the  following  two 
primary  factors: 

a.  Determination  of  the  readiness  of  own  combat  power  which  is  achieved  through  the 
evaluation  of:  availability  of  combat  power  (e.g.,  current  combat  power  inventory, 
assignment  status,  readiness  of  support  resources,  and  damage  to  combat  power 
resources),  and  the  reliability  of  combat  power  resources  (e.g.,  mean  time  between 
failures).  This  assessment  is  completed  independent  of  the  threat. 

b.  Performance  prediction  of  combat  resources  which  is  dependent  on  the  reliability  of 
combat  power  resources,  damage  to  combat  power  deployment  systems,  performance 
of  combat  power  resource  (e.g.,  lethality,  probability  of  success,  time  constraints 
etc.),  and,  impact  of  environmental  constraints  (e.g.,  physical  obstructions,  electronic 
attack,  weather  conditions,  sea  state  etc.)  on  CP  deployment.  This  goal  is  dynamic 
and  dependent  of  the  threat. 

The  output  is  a  list  of  combat  power  deployment  options  for  each  threat,  with  the  associated 
degrees  of  freedom  (e.g.,  time  and  range).  This  list  of  available  options  against  each  single  threat 
is  maintained  with  consideration  of  combined  effects,  synergy  and  usage  constraints. 
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Engageability  List 


Figure  4:  Engageability  Assessment  [10] 


2.6  Combat  Power  Management 

The  combat  power  management  can  be  decomposed  into  three  phases  (Figure  5)  [10]: 

1 .  Response  planning  ensures  that  one  or  more  combat  power  resources  are  assigned  to  engage 
each  target  (i.e.,  atomic  actions),  including  the  assignment  of  supporting  resources  (e.g., 
sensors,  communications).  This  involves  assignment  of  both  resources  (i.e.,  pure  allocation 
problem)  and  start  and  end  times  to  activities  (i.e.,  pure  scheduling  problem).  In  addition, 
response  planning  encompasses  joint  resource  allocation  and  scheduling  problems  that 
generate  a  ranked  engagement  list  of  the  targets  for  the  response  execution.  A  response  (or 
engagement  plan)  is  a  coordinated  (conflict-free)  schedule  (timeline)  for  the  application  of 
the  selected  CP  components. 

2.  Response  Execution  involves  executing  in  real-time  the  coordinated  scheduled  plan  for  the 
application  of  CP  resources  to  counter  targets  within  the  current  tactical  situation. 


DRDC  Toronto  CR  2009-041 


3.  Response  Monitoring  is  required  since  the  responses  are  executed  in  a  dynamic 
environment,  subject  to  uncertainty,  changing  goals,  and  changing  conditions.  As  such,  the 
actual  execution  contexts  will  be  different  from  the  projected  ones,  i.e.,  the  ones  that 
motivated  the  construction  of  the  original  response.  Monitoring  is  essential  to  help  detect, 
identify  and  handle  contingencies  caused  by  uncertainty  and  the  changing  nature  of  the 
environment.  Also  involved  is  the  evaluation  of  the  outcome  of  the  executed  actions.  This 
boils  down  to  performing  damage  assessment  (e.g.,  capability)  of  engaged  target(s)  and 
assessing  the  damage  inflicted  to  own-assets  by  opponent  forces. 


Figure  5:  Application  of  Combat  Power  [10] 


Combat  power  atomic  actions  can  be  grouped  into  three  categories  [12]: 

1.  Deterrence  employs  a  series  of  measures  such  as  manoeuvring  platforms  and  radio  warnings 
in  order  to  convince  a  target  that  the  consequences  of  coercion  or  armed  conflict  would 
outweigh  the  potential  gains; 

2.  Softkill  measures  attempt  to  defeat  a  target  through  the  use  of  deception,  seduction,  or 
confusion  methods.  Included  is  the  use  of  chaff  for  seduction,  distraction  and  re-acquisition 
as  well  as  jamming;  and 
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3.  Hardkill  measures  include  all  weapons  and  armaments  to  achieve  the  physical  destruction  of 
a  target.  These  include  surface-to-surface  missiles  (SSM),  surface-to-air  missiles  (SAM), 
guns,  and  torpedoes. 

2.7  INCOMMANDS  Command  Decision  Support  Capability 

The  INCOMMANDS  decision  support  system  is  a  series  of  single-role  displays  that  present 
information  and  support  the  necessary  interactions  required  by  each  combat  role  in  order  to 
support  decision-making  from  an  individual  perspective.  An  illustration  of  a  conceptual 
INCOMMANDS  prototype  comprising  multiple  single-role  displays  is  provided  in  Figure  6.  The 
console  is  comprised  of  four  interdependent  types  of  displays:  Threat  Evaluation  functional  view. 
Combat  Power  Management  functional  view,  physical  view,  and  general  purpose.  The 
complementary  advantages  provided  by  each  display  are  envisioned  to  augment  the  operator’s 
decision-making  capabilities.  Currently  only  the  TE,  CPM,  and  physical  displays  have  been 
implemented. 


Multi-Purpose 

Display 


Combat  Power 
Management 
Functional  Display 


Physical  Display 


Threat  Evaluation 
Functional  Display 


Figure  6:  Conceptual  Console  for  Single-Role  Displays 


The  TE  functional  display  is  comprised  primarily  of  a  2-dimensional  threat  list.  The  list  presents  a 
relative  ranking  of  threats  based  on  their  threat  rating  as  calculated  by  the  CDSC  using  the 
appropriate  TE  algorithm.  To  that  end,  the  operator  is  able  to  monitor  the  complete  threat  picture 
as  well  as  perform  pre-emptive  planning  as  required  to  avoid  a  potential  engagement  (Figure  7). 
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Figure  7:  Threat  Evaluation  Functional  View 


The  CPM  functional  view  is  aimed  at  supporting  the  management  and  application  of  combat 
power  through  a  series  of  decision  aids.  The  CPM  functional  view  consists  of  four  predominant 
groupings:  target-based  view  of  engagement  plans,  resource-based  view  of  engagement  plans, 
engagement  drill  downs,  and  notifications.  Figure  8  provides  a  depiction  of  the  CPM  functional 
view. 
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Figure  8:  Combat  Power  Management  Functional  View 
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3  Evaluation  Methodology 


3.1  General 

Regardless  of  the  level  of  analysis  completed  in  support  of  designing  an  interface,  experience  has 
shown  that  particular  problems  only  appear  when  the  design  is  thoroughly  evaluated. 
Consequently,  evaluating  the  OMl  design  is  key  to  a  product’s  success.  To  that  end,  OMl 
evaluation  has  three  main  goals: 

1.  to  assess  the  extent  of  the  system’s  functionality 

2.  to  assess  the  effect  of  the  interface  on  the  user 

3.  to  identify  problems  (a.k.a.  usability  bugs) 

Ideally,  OMl  evaluation  should  occur  early  and  often  during  the  design  process.  Early  on,  OMl 
assessments  are  employed  by  the  HE  analyst  to  gather  and  validate  data  on  how  operators  conduct 
their  work.  As  the  OMl  design  progresses,  assessments  provide  valuable  input  for  analyzing 
initial  design  concepts  and,  in  the  later  stages,  to  test  specific  tasks. 

Research  in  Human  Computer  Interaction  (HCl)  has  developed  a  multitude  of  evaluation 
techniques  for  analyzing  and  subsequently  improving  an  OMl.  Each  methodology  highlights 
different  usability  issues;  therefore,  HE  analysts  can  choose  and  mix  techniques  to  fit  the  needs 
and  nuances  of  the  design  situation.  In  fact,  testing  with  different  techniques  provides 
complementary  results.  A  good  evaluation  process  means  major  problems  are  caught  early  with 
minor  problems  ironed  out  as  the  OMl  is  being  refined.  It  is  easier  and  more  cost  effective  to 
make  product  changes  earlier  rather  than  later  in  the  design  process. 

Three  categories  of  evaluation  methods  are: 

1.  User  Observations.  User  observations  are  conducted  in  a  lab  using  a  representative  sample 
of  the  eventual  users  performing  tasks  that  depict  how  the  system  will  be  employed  in  the 
“real”  world.  Evaluators  uncover  problems  by  observing  the  participants  completing  the 
tasks.  Two  techniques  are: 

a.  Controlled  experiments.  Used  to  prove  that  varying  an  independent  variable  causes 
the  effect  observed  in  the  dependent  variable.  Rigorous  control  ensures  that 
uncontrolled  variables  do  not  affect  the  results  and  their  interpretation.  Statistical 
analysis  allows  inferences  to  be  made  concerning  causal  relationships  between  the 
manipulation  of  the  independent  variables  and  changes  in  the  dependent  variables. 

b.  Usability  testing.  End  users  complete  a  set  of  real  tasks  with  a  prototype.  HE 
analysts  observe  their  performance  and  collect  empirical  data  (e.g.,  errors  made, 
difficulties  experienced,  and  workload  measures). 

2.  Field  Studies.  Field  studies  help  to  address  a  problem  with  evaluating  a  product  in  the  lab: 
the  failure  to  account  for  conditions,  context,  and  tasks  central  to  a  product’s  real  world  use. 
Field  studies  involve  the  HE  analyst  studying  systems  in  use  on  real  tasks  in  real  work 
settings.  Two  techniques  are: 
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a.  Ethnography,  Based  on  the  premise  that  human  activities  are  socially  organized; 
therefore,  ethnography  looks  at  patterns  of  collaboration  and  interaction.  The 
ethnographer  gathers  data  by  passively  observing  and  recording  participants  as  they 
work  in  their  natural  environment  using  the  available  technology  and  tools.  This 
includes  focusing  on  social  relationships  and  their  effect  on  the  nature  of  work.  The 
goal  is  to  identify  routine  practices,  problems,  and  development  possibilities  within  a 
given  activity  or  setting. 

b.  Contextual  Inquiry.  Employs  an  interview  methodology  to  gain  knowledge  of  what 
users  do  in  their  real  world  context.  HE  analysts  conduct  interviews  by  observing  and 
talking  with  target  users  as  they  work  in  order  to  record  user  tasks  and  dialogue, 
breakdowns  and  workarounds,  and  artefacts  employed  to  help  achieve  work  goals. 

The  objective  is  to  understand  the  users’  motivations  and  strategies  with  the  objective 
of  creating  representations  of  the  work. 

3.  Interface  inspections.  Interface  inspections  involve  HE  analysts  inspecting  an  OMl  for 
usability  problems  according  to  a  set  of  criteria,  usually  related  to  how  individuals  see  and 
perform  a  task.  These  methods  use  judgment  as  a  source  of  feedback  when  evaluating  the 
system.  Techniques  include: 

a.  Heuristic  evaluation,  HE  analyst(s)  visually  inspect  the  system  and  judge  its 
compliance  with  recognized  usability  principles  (heuristics).  Non-compliant  aspects 
are  captured  by  describing  the  problem,  its  severity,  and  possible  fixes. 

b.  Cognitive  walkthroughs,  HE  analysts  work  through  a  task,  step  by  step,  and 
identify  potential  problems  against  psychological  criteria.  For  each  task,  the  analyst 
considers  the  interaction’s  impact  on  the  user,  the  cognitive  processes,  and  potential 
learning  problems.  For  each  action,  data  related  to  the  users’  goals,  tasks,  knowledge, 
the  interface’s  visible  state,  and  the  relationships  among  these  factors  are  recorded. 

The  goal  is  to  identify  task  sequences  that  may  cause  difficulties  with  learning  the 
system. 

c.  Task  centered  walkthroughs,  HE  analyst(s)  step  through  scenarios  with  tasks  and 
user  descriptions.  At  each  step  they  question  whether  the  user  has  the  knowledge  to 
do  the  step  and  whether  it  is  believable  that  the  user  would  actually  do  it.  If  the 
answer  to  either  question  is  no,  then  a  usability  problem  is  reported. 

d.  Pluralistic  walkthroughs,  HE  analyst(s)  present  participants  with  a  hardcopy  of  the 
first  screen  encountered  in  a  scenario.  Participants  independently  write  down  the 
actions  they  would  perform  to  complete  the  specified  task.  Next,  the  “right”  answer  is 
announced.  The  participants  verbalize  their  responses  and  discuss  potential  usability 
problems  due  to  “incorrect”  answers.  The  process  continues  until  the  scenario  is 
complete. 

Evaluation  early  in  the  design  stage  (prior  to  any  implementation)  tends  to  be  analytical  involving 
design  experts.  As  the  design  is  slowly  implemented,  end  users  are  brought  into  the  evaluation  as 
subjects.  As  vital  as  the  analytical  techniques  are  for  filtering  and  refining  the  design,  they  are  not 
a  replacement  for  usability  testing.  Testing  of  this  nature  should  be  done  at  least  once  with  end 
users  performing  representative  tasks  that  the  product  has  been  designed  to  support. 
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3.2  INCOMMANDS  TDP  Evaluation  Framework 

The  evaluation  framework  for  the  INCOMMANDS  TDP  involved  a  two  step  process  to  identify 
potential  usability  problems  with  the  OMl  design: 

1.  Heuristic  Evaluation:  Specific  elements  of  the  INCOMMANDS  CDSC  that  are  not 
compliant  with  the  INCOMMANDS  TDP:  Human  Factors  Design  and  Evaluation  Guide  [2] 
were  identified  and  recorded  in  a  compliance  matrix.  Non  compliant  items  were  subsequently 
assigned  a  major  or  minor  severity  rating. 

2.  Usability  and  Utility  Testing:  The  purpose  of  the  trial  was  to  provide  user  feedback  for  ease 
of  use  and  utility  to  guide  modifications  for  the  future  development  cycles  of  the 
INCOMMANDS  CDSC  OMl.  In  preparation  for  these  trials,  an  experimental  protocol  was 
generated  and  subsequently  presented  and  accepted  by  the  DRDC  Human  Research  Ethics 
Committee  (HREC). 

These  two  evaluation  techniques  were  proposed  as  part  of  the  INCOMMANDS  TDP  since  their 
findings  are  complementary.  As  previously  mentioned,  the  Heuristic  Evaluation  ensures  that  the 
INCOMMANDS  CDSC  adheres  to  industry-accepted  HCl  principles  and  guidelines.  The 
Usability  and  Utility  Testing  is  not  conducive  to  capturing  these  types  of  OMl  issues.  By 
including  operational  participation  during  the  Usability  and  Utility  Testing,  the  ability  of  the 
INCOMMANDS  CDSC  to  satisfy  the  end  user’s  goals  and  tasks  can  be  assessed — which  is  not 
feasible  via  a  Heuristic  Evaluation. 

These  two  evaluation  techniques  were  chosen  for  the  INCOMMANDS  TDP  in  favour  of  the  other 
available  options  for  several  reasons.  For  instance,  field  studies  (i.e.,  ethonography,  contextual 
inquiry)  require  observing  the  operators  performing  their  tasks  in  their  native  environment. 
Access  to  the  Naval  operators  conducting  threat  evaluation  and  combat  power  management 
activities  on-board  the  HAEIFAX  Class  frigate  in  a  realistic  environment  is  not  easily  achieved. 
With  respect  to  the  available  interface  inspection  techniques,  a  Heuristic  Evaluation  was 
performed  as  opposed  to  a  Cognitive  Walkthrough  since  the  INCOMMANDS  CDSC  is  intended 
to  be  employed  by  trained  operators;  therefore,  the  leamability  of  the  system  was  not  an  area  of 
investigation.  Furthermore,  the  GUI  for  the  INCOMMANDS  CDSC  was  developed  based  on 
goals  and  tasks  identified  through  interactions  with  the  Naval  operators.  Other  walkthrough 
techniques  (Task  Based  and  Pluralistic)  were  not  employed  in  favour  of  the  Usability  and  Utility 
Testing  involving  the  operational  community.  The  functionality  embedded  in  the  current 
implementation  of  the  INCOMMANDS  CDSC  was  also  not  conducive  to  facilitating  a 
walkthrough  since  the  operators  were  limited  to  monitoring  system-generated  assessments  with 
minimal  available  interactions. 

The  following  sections  provide  an  overview  of  the  methodology  utilized  for  Heuristic  Evaluation 
and  Usability  and  Utility  Testing  of  the  INCOMMANDS  CDSC. 

3.2.1  Heuristic  Evaiuation 

3.2. 1.1  General 

There  exist  numerous  principles  fundamental  to  the  design  and  implementation  of  effective 
interfaces  for  traditional  OMl  environments.  These  principles  manifest  in  the  form  of  guidelines 
and  heuristics  which  can  be  differentiated  as  follows: 
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1 .  Heuristics  are  the  motherhood  rules  and  principles  that  describe  common  properties  of  usable 
interfaces.  They  are  generally  more  abstract  than  traditional  guidelines.  Consequently, 
possessing  U1  design  knowledge  and  experience  helps  to  accurately  understand  and  interpret 
them. 

2.  Guidelines  can  be  construed  as  good  practices  within  a  general  design  domain  (such  as 
Windows  in  the  case  of  the  Microsoft  Solutions  product  portfolio).  Holistically,  they  are 
based  broadly  on  the  usability  heuristics.  Specifically,  they  provide  useful  low-level  guidance 
on  the  design  of  usable  interfaces  in  areas  such  as  control  design,  branding  elements,  and 
window  behaviour.  Since  they  are  generally  more  specific  than  heuristics,  less  design 
knowledge  and  experience  is  required  to  understand,  interpret,  and  apply  them. 

3.2. 1.2  Procedure 

Numerous  sources  exist  for  describing  heuristics  and  guidelines  that  could  be  used  to  inform  the 
design  of  the  INCOMMANDS  CDSC  OMl.  The  INCOMMANDS  CDSC  OMl  design  effort 
leveraged  the  guidance  set  forth  by  the  INCOMMANDS  TDP:  Human  Factors  Design  and 
Evaluation  Guide  [2].  This  guide  is  an  amalgamation  of  the  following  two  reports: 

1.  Command  Decision  Aid  Technology  (COMDAT)  OMl  Style  Guide  [9).  The  COMDAT 
OMl  Style  Guide  was  created  to  provide  a  common  framework  for  new  developments  in 
Command  Decision  Aids.  The  style  guide  is  designed  to  be  used  for  all  Maritime  Command 
and  Control  systems  and  upgrades.  The  objective  is  to  provide  a  common  look  and  feel  that  is 
compatible  with  existing  systems,  yet  accommodates  new  developments  and  knowledge. 

2.  Development  of  Decision  Aid  Implementation  Guidance  for  the  INCOMMANDS 
Human  Factors  Design  and  Evaluation  Guide  [8].  This  document  is  an  extension  to  the 
aforementioned  COMDAT  OMl  Style  Guide  and: 

a.  incorporates  recommended  standards  and  guidelines  that  informs  the  OMl  design  and 
decision  aiding  concepts  developed  within  the  INCOMMANDS  TDP  to  ensure 
consistency  with  Human  Factors  best-practice; 

b.  provides  a  common  OMl  design  guidance  for  existing  decision  aids,  decision  aids 
under  development,  and  future  decision  aiding  concepts,  within  the  context  of 
Maritime  C2,  including  TE  and  CPM;  and, 

c.  provides  general  guidance,  in  terms  of  suggested  metrics  and  tools,  for  the  evaluation 
of  a  proposed  OMFs  compliance  with  the  guidelines  within  the  style  guide. 

The  heuristic  evaluation  identified  specific  elements  of  the  INCOMMANDS  CDSC  that  are  not 
compliant  with  the  INCOMMANDS  Human  Factors  Design  and  Evaluation  Guide.  It  was 
performed  by  a  HF  analyst  from  CAE  PS  with  results  captured  in  a  compliance  matrix  with  the 
following  columns: 

1.  Compliance.  The  Compliance  column  places  each  design  feature  in  one  of 
three  categories  as  follows: 

a.  Not  compliant.  The  INCOMMANDS  CDSC  does  not  satisfy  the  objectives  of  the 
stated  guideline; 
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b.  Out  of  Scope  -  Experimental.  Compliance  to  the  guideline  could  not  be  evaluated 
via  inspection.  It  was  anticipated  that  the  guideline  could  be  verified  through 
usability  testing  or  experimentation;  and 

c.  Out  of  Scope.  Design  features  that  are  not  implemented  in  the  current 
INCOMMANDS  CDSC  build.  Any  instances  in  which  the  Scientific  Authority  and 
HF  analyst  agree  that  an  element  identified  in  the  compliance  matrix  is  out  of  the 
scope  of  the  current  INCOMMANDS  CDSC,  that  design  element  should  be 
identified  and  reserved  for  consideration  in  future  development  of  the 
INCOMMANDS  CDSC. 

2.  Paragraph,  Column  two  identifies  the  paragraph  number  of  the  guideline  INCOMMANDS 
Human  Factors  Design  and  Evaluation  Guide  (Version  1.0)  that  is  relevant  to  the  design 
feature. 

3.  INCOMMANDS  HF  Design  and  Evaluation  Guide.  Column  3  contains  the  text  of  the 
guideline  that  is  relevant  to  the  design  feature. 

4.  INCOMMANDS  CDSC,  Column  4  describes  each  identified  design  feature  of  the 
INCOMMANDS  CDSC  OMl. 

5.  Severity  Rating.  Column  5  describes  the  severity  of  the  non-compliance.  A  minor  non- 
compliance  is  deemed  to  have  minimal  impact  on  the  ability  of  the  operator  to  complete  the 
task  at  hand.  A  major  non-compliance  is  considered  to  severely  impact  the  ability  of  the 
operator  to  complete  the  task  at  hand. 

3.2.2  Usability  and  Utility  Testing 

3.2. 2.1  General 

Although  measures  of  system  usability  do  not  speak  directly  to  the  issue  of  decision  making 
quality,  they  serve  as  a  diagnostic  tool  for  highlighting  potential  OMl  problems  that  may  have 
interfered  with  a  participant’s  task  relevant  performance.  In  turn,  OMl  problems  can  result  in 
poor  decision  making  on  the  part  of  the  operator.  From  the  perspective  of  the  participant, 
usability  is  important  because  it  can  make  the  difference  between  performing  a  task  accurately 
and  completely  or  not  at  all.  From  the  perspective  of  the  INCOMMAND  TDP,  usability  is 
important  because  it  can  mean  the  difference  between  the  success  or  failure  of  a  system;  poor 
usability  can  reduce  the  operational  effectiveness  and  integrity  of  the  human-machine  system.  In 
all  cases,  lack  of  usability  can  cost  time  and  effort,  and  can  greatly  determine  the  success  or 
failure  of  a  system.  Assessing  the  utility  of  the  system  is  essential  because  it  is  important  to  know 
if  the  INCOMMANDS  CDSC  helps  or  hinders  the  operators’  ability  to  do  their  job.  A 
combination  of  questionnaires  and  interviews  were  used  to  collect  usability  and  utility  data  from 
each  participant. 

The  purpose  of  the  Usability  and  the  Utility  Trial  of  the  INCOMMANDS  CDSC  was  to  elicit 
feedback  from  Naval  operators  (ORO  and  SWC)  in  order  to  identify  potential  usability  problems 
to  be  fixed  with  future  builds  as  well  as  ascertain  its  ability  to  assist  Naval  operators  with  the 
conduct  of  TE  and  CPM  functions  on-board  a  Halifax  Class  Frigate.  Furthermore,  the  testing 
investigated  the  impact  that  the  new  decision  support  system  could  have  on  operators’  situation 
awareness  and  workload. 
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Through  their  involvement  in  the  study,  operators  were  able  to  contribute  to  the  development, 
validation  and  implementation  of  decision  support  concepts  to  support  the  conduct  of  TE  and 
CPM  functions  by  CF  personnel  on-board  a  HALIFAX  Class  frigate. 

The  following  sections  outline  the  Usability  and  Utility  Testing  structured  methodology  that  was 
used  to  assess  development,  validation  and  implementation  of  decision  aid  concepts  designed  as 
part  of  the  INCOMMANDS  CDSC. 

3.2. 2.2  Approach 

The  Technology  Acceptance  Model  (TAM)  was  utilized  as  an  approach  for  structuring  the 
Usability  and  Utility  Testing.  The  TAM  is  an  information  systems  theory  that  models  how  users 
come  to  accept  and  use  a  technology.  The  model  suggests  that  when  users  are  presented  with  a 
system,  a  number  of  factors  will  influence  their  decision  about  how  and  when  they  will  use  it, 
notably: 

1 .  Perceived  Utility  (i.e.  usefulness)  is  defined  as  ‘the  degree  to  which  a  person  believes  that 
using  a  particular  system  would  enhance  his  or  her  job  performance’  [6]. 

2.  Perceived  Ease-of-Use  is  defined  as  ‘the  degree  to  which  a  person  believes  that  using  a 
particular  system  would  be  free  from  effort’  [6]. 

The  TAM  utilizes  a  questionnaire  that  has  been  assessed  for  robustness  across  populations  and 
predictive  validity.  Studies  have  shown  high  reliability  and  good  test-retest  reliability  and  that  the 
instrument  had  predictive  validity  for  intent  to  use,  self-reported  usage  and  attitude  toward  use. 
The  sum  of  this  research  has  confirmed  the  validity  of  this  instrument,  and  supports  its  use  with 
different  populations  of  users  and  different  software  choices. 

3.2. 2.3  Assessments  of  System  Usability  and  Utility 

To  assess  operator  perceptions  of  the  usability  of  the  INCOMMANDS  CDSC,  questionnaires 
relating  to  the  high-level  usability  aspects  of  both  the  CPM  and  the  TE  (e.g.,  suitability  of  screen 
design)  were  developed  and  administered.  Ratings  are  based  on  a  five -point  Likert  scale;  ranging 
from  1:  Strongly  Disagree  to  5:  Strongly  agree.  A  sample  question  is  provided  below.  Complete 
usability  questionnaires  for  TE  and  CPM  functional  views  can  be  found  in  Annex  C  and  Annex  D 
respectively. 


1 .  The  size  of  the  threat  list  is  suitable. 


@ 

© 

Suggested 

Strongly 

Disagree 

Border 

Agree 

Strongly 

Improvements 

Disagree 

Agree 

Similarly  a  questionnaire  relating  to  the  perceived  utility  of  the  workstation  functions  and 
capabilities  was  administered  to  assess  operator  perceptions  of  system  utility.  Ratings  are  based 
on  a  five -point  Likert  scale;  ranging  from  1:  Strongly  Disagree  to  5:  Strongly  agree.  A  sample 
question  is  provided  below.  The  complete  utility  questionnaire  can  be  found  in  Annex  E. 
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1.  The  Threat  Evaluation  View  provides  me  with  a  more  comprehensive  assessment  of  all 
threats  in  the  environment. 


@ 
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Suggested 

Strongly 

Disagree 

Border 

Agree 

Strongly 

Improvements 

Disagree 

Agree 

In  addition  to  measuring  the  utility  of  the  INCOMMANDS  CDSC,  the  complete  utility 
questionnaires  included  a  number  of  questions  that  queried  the  operators  about  their  perceived 
situation  awareness  and  workload.  The  Situation  Awareness  Rating  Technique  (SART)  was  also 
administered  to  assess  operators’  perceptions  of  their  perceived  situation  awareness.  Operators 
did  not  answer  all  the  questions;  therefore,  scores  for  the  SART  were  not  calculated.  Initially,  the 
intent  was  to  investigate  and  assess  the  operator’s  trust  with  the  INCOMMANDS  CDSC; 
however,  this  data  was  not  collected  in  favour  of  focusing  on  the  areas  of  utility,  usability, 
situation  awareness,  and  workload. 

3.2. 2.4  Equipment  and  Facilities 

The  Usability  and  Utility  Testing  was  conducted  at  the  Canadian  Forces  Maritime  Warfare  Centre 
in  Halifax,  Nova  Scotia.  The  equipment  employed  was  a  portable  version  of  the  INCOMMANDS 
Command  Decision  Support  (CDS)  Laboratory.  In  addition,  video  and  audio  recordings  were 
captured  of  both  the  participants  and  the  presentation  of  information  on  the  individual  displays 
during  each  trial.  As  such,  verbal  dialogues  and  gestures  could  be  correlated  with  the  data  being 
presented  on  the  screen. 

The  INCOMMANDS  CDS  Lab  used  for  the  testing  was  not  fully  functional  and  therefore,  did  not 
allow  the  operator  to  interact  fully  with  the  information  on  either  the  TE  or  the  CPM  Displays. 
However,  operators  indicated  that  they  were  able  to  understand  how  the  INCOMMANDS  system 
could  potentially  augment  their  decision  making  capabilities. 

3.2. 2.5  Scenarios 

Scenarios  were  developed  for  the  practice  and  test  sessions  using  STAGE  Scenario.  The  practice 
scenario  was  a  15-minute  low-complexity  scenario  with  a  small  number  of  mid-  and  high-level 
threats  and  only  two  engagements.  The  trial  scenario  was  a  20-minute  high-complexity  scenario 
that  contained  a  number  of  medium-  and  high-level  threats  as  well  as  five  engagements. 

3.2. 2.6  Procedure 
3.2. 2.6.1  General 

The  complete  Usability  and  Utility  Testing  transpired  over  three  days;  however,  the  total  time 
commitment  for  each  participant  was  approximately  2  hours.  The  following  sections  outline  the 
procedure  utilized  as  part  of  the  Usability  and  Utility  Testing. 
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3.2. 2.6.2  Orientation  Briefing 


Following  a  brief  introduction  to  each  member  of  the  experimental  staff,  the  participants  were 
given  an  overview  of  the  general  purpose  of  the  trial,  and  were  then  given  the  opportunity  to  ask 
questions  before  signing  the  consent  form  (Annex  B).  After  signing  the  consent  form,  participants 
were  asked  to  provide  some  professional  and  personal  background  information  (e.g.,  number  of 
years  of  experience  as  an  operator).  Training  of  the  participant  and  familiarization  with  the 
INCOMMANDS  CDSC  followed. 

3.2. 2.6.3  Training  Session 

In  the  training  session,  operators  viewed  the  low  complexity  scenario  and  they  were  encouraged 
to  interact  with  the  system  and  to  ask  questions.  At  the  end  of  the  training  session,  operators  were 
asked  if  they  had  any  questions  about  the  system.  Once  all  questions  were  answered,  the  usability 
and  utility  trial  scenario  began. 

3.2. 2.6.4  Test  Session 

Operators  were  seated  in  the  experimental  testing  room  in  front  of  the  fNCOMMANDS  CDSC. 
Operators  were  given  a  detailed  briefing  by  the  SME,  whose  primary  role  was  to  act  as  the  liaison 
between  the  experimental  staff  and  the  operators.  The  SME  was  an  experienced  Naval  operator 
who  is  familiar  with  the  decision  support  concepts  and  the  functional  displays  of  the 
INCOMMANDS  CDSC  prototype. 

Following  any  questions  the  operator  had  regarding  the  information  covered  in  the  briefing, 
operators  began  the  usability  and  utility  trial.  During  the  trial,  participants  were  encouraged  to 
provide  subjective  feedback  about  the  quality  of  various  aspects  of  the  simulation  environment 
and  to  comment  on  any  ideas  they  might  have  about  what  would  be  a  useful  but  currently  not 
implemented  in  the  physical  and  functional  displays.  The  operator’s  responses  were  recorded 
(both  audio  and  video).  Immediately  following  the  trial,  each  operator  completed  a  threat 
evaluation  and  combat  power  usability  and  utility  questionnaire.  Following  completion  of  the 
trial,  the  operators  were  debriefed  by  the  experimental  staff. 
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4  Heuristic  Evaluation 


4.1  General 

The  Heuristic  Evaluation  of  the  INCOMMANDS  CDSC  OMl  was  aimed  at  identifying  non- 
compliances  with  the  guidelines  set  forth  by  the  fNCOMMANDS  TDP:  Human  Factors  Design 
and  Evaluation  Guide  [2],  The  results  are  detailed  in  the  compliance  matrix  located  in  Annex  A. 
The  matrix  is  only  comprised  of  non-compliant  guidelines  for  which  the  system  does  not  satisfy 
their  criteria;  therefore,  compliant  guidelines  are  not  present.  This  section  provides  an  overview 
of  these  results  and  conclusions  that  can  be  drawn  based  on  this  evaluation. 

4.2  Results 

As  captured  in  Annex  A,  the  Heuristic  Evaluation  uncovered  numerous  design  non-compliances 
with  the  OMl  guidelines  for  the  current  build  of  the  fNCOMMANDS  CDSC  OMl.  A  significant 
portion  of  the  non-compliant  guidelines  are  classified  as  Major  (i.e.,  significant  obstacle  for  the 
operator  to  complete  the  task  at  hand  or  meet  the  objectives  of  the  goal).  The  majority  of  items 
that  were  classified  as  Major  non-compliancy  issues  are  related  to  following  categories  of 
guidelines: 

1 .  Decision  support  (Sections  7  and  8).  These  guidelines  are  aimed  at  keeping  the  operators 
‘in  the  loop’.  For  example,  the  existing  fNCOMMANDS  CDSC  OMl  does  not  provide 
operators  with  alternative  action  plans  or  allow  operators  to  select  their  choice  of  action 
plans. 

2.  Interactive  control  guidelines  (Section  12,5).  These  guidelines  provide  the  user  the  ability 
to  remain  ‘in  control’  while  providing  appropriate  feedback.  The  current  implementation  of 
the  fNCOMMANDS  CDSC  OMl  performs  the  engagement  assessments  automatically 
without  an  explicit  action  required  on  the  part  of  by  the  operator.  As  such,  the  system  is  “in 
control”  as  opposed  to  the  operator. 

3.  Data  query  (Section  12.3.3).  Operators  are  not  currently  able  to  query  the  INCOMMANDS 
CDSC  in  order  to  retrieve  and  display  data. 

Many  of  the  decision  support  guidelines  will  need  to  be  evaluated  further  through  other 
evaluation  techniques.  For  example,  experiments  can  be  used  to  gather  objective  measures  of  the 
adequacy  of  OMl  ease  of  use  and  useful  as  well  as  assess  the  adequacy  of  the  operator’s  SA  and 
workload. 

The  large  number  of  non-compliances  may  be  viewed  as  disconcerting.  Development  work 
completed  to  date  on  the  INCOMMANDS  CDSC  has  focused  on  establishing  a  functional 
version  of  the  prototype  in  order  to  investigate  technological  and  operational  concepts.  This 
prioritization  of  effort  was  in  response  to  schedule  and  budget  constraints.  As  such,  many  of  the 
HCl  specific  concerns  for  ensuring  a  usable  end  version  were  not  addressed  during  the 
development  phase  of  the  project.  However,  the  requirements  in  the  Human  Factors  Design  and 
Evaluation  Guide  [2]  are  extremely  important  to  ensure  compliance  with  industry-accepted  HCl 
best  practices.  As  such,  these  non-compliances  will  need  to  be  addressed  if  future  versions  of  the 
fNCOMMANDS  CDSC  are  developed.  Subsequent  Heuristic  Evaluations  will  also  need  to  be 
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conducted  to  ensure  these  versions  are  compliant  with  the  guidelines  outlined  in  the  Human 
Factors  Design  and  Evaluation  Guide  [2]. 

Furthermore,  a  number  of  guidelines  stated  in  the  evaluation  guide  were  not  applicable  to  the 
current  version  of  the  INCOMMANDS  CDSC  OMl.  Examples  include  the  lack  of  particular 
controls  (e.g.,  menu  options)  or  other  features  such  as  maps  and  graphical  displays.  If  in  the 
future  a  more  complete  development  of  the  INCOMMANDS  CDSC  is  undertaken,  it  may  be 
helpful  for  operators  to  have  features,  such  as  the  ones  previously  highlighted,  installed. 
Therefore,  it  is  important  that  the  Human  Factors  Design  and  Evaluation  Guide  [2]  continue  to 
guide  any  future  versions  of  the  INCOMMANDS  CDSC. 

Finally,  compliance  of  the  INCOMMANDS  CDSC  to  numerous  applicable  guidelines  stated  in 
the  Human  Factors  Design  and  Evaluation  Guide  [2]  could  not  be  confirmed  via  an  inspection 
technique  such  as  the  Heuristic  Evaluation  methodology.  These  guidelines  have  been  identified 
and  should  be  verified  through  another  complementary  HCl  evaluation  technique  such  as  a 
controlled  experiment. 

4.3  Conclusions 

The  results  from  the  Heuristic  Evaluation  provide  specific  direction  to  support  future 
development  of  the  INCOMMANDS  CDSC  in  order  to  improve  the  overall  usability  of  the 
system.  As  such,  any  additional  work  to  be  completed  on  the  system  should  attempt  to  improve 
the  usability  of  the  system  by  both  fixing  identified  non-compliances  as  well  as  adding  missing 
functionality. 

Finally,  there  exist  additional  design  aspects  that  are  not  covered  by  the  current  set  of  guidelines 
captured  in  the  INCOMMANDS  TDP:  Human  Factors  Design  and  Evaluation  Guide  [2]  (e.g., 
transparent  overlays).  Guidelines  will  need  to  be  researched  and  included  in  the  next  version  of 
the  document  for  future  implementations  of  this  type  of  OMl  element. 
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5  Usability  and  Utility  Testing  Results 


5.1  General 

Results  stemming  from  the  Usability  and  Utility  Testing  are  presented  in  the  following  sections. 
These  results  have  been  broken  down  according  to  the  following  groupings  for  both  the  TE  and 
CPM  functional  views: 

1.  Usability  -  Display  of  information;  Usefulness  of  the  information;  Participants  comments 

2.  Utility  -  Overall  assessment  of  threats  and  engagement  plans;  Impact  on  workload,  situation 
awareness  and  operations  room  communications 

Furthermore,  an  overall  assessment  of  impact  of  the  TE  and  CPM  functional  views  on  perceived 
workload  and  situation  awareness  is  presented. 

5.2  Participants 

Six  active  and  qualified  CF  Naval  operators  between  37  and  50  years  (M  =  46  years)  were 
recruited  to  participate  in  this  trial.  Operators  were  either  qualified  as  ORO  or  SWC  with  five 
OROs  and  one  SWC  participating.  The  ORO  and  the  SWC  are  the  two  primary  ship  combat 
operators  (i.e.,  they  are  in  charge  of  threat  evaluation,  engageability  assessment,  combat  power 
management  tasks)  within  the  operations  room  of  the  HAEIFAX  Class  frigate.  The  ORO  is 
responsible  for  ensuring  that  the  appropriate  response  is  taken  to  counter  any  threat  encountered 
by  the  ship.  As  such,  the  ORO  evaluates  the  requirements  for  resources  of  each  warfare  area  (air, 
surface,  subsurface);  resolving  any  conflicts  that  arise,  to  most  effectively  allocate  those  resources 
when  conducting  multi-warfare  operations.  The  SWC  manages  two  warfare  areas  (surface  and 
air)  including  the  assessment  of  threats,  prioritization  of  targets,  performance  prediction  of  the 
above-water  weapons  and  sensors  as  well  as  the  planning  and  execution  of  response. 

All  operators  had  experience  with  the  existing  Command  and  Control  System  (CCS)-330. 
Participating  operators  were  recruited  from  Halifax,  N.S.  and  conducted  the  tasks  while  on  duty; 
however,  participation  was  voluntary. 

5.3  Usability  Questionnaire  Results 

5.3.1  TE  Functional  View 

5.3. 1.1  General 

Questionnaire  rating  data  was  collated  and  summary  statistics  (mean  and  standard  deviation)  were 
calculated.  Mean  questionnaire  results  (N  =  6)  and  standard  deviations  (eyelashes)  for  the  Threat 
Evaluation  functional  display  system  are  shown  in  Figure  9  and  Figure  10.  The  following 
sections  highlight  the  participants’  feedback  as  gathered  through  the  questionnaires. 
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5.3.1. 2  Display  of  Information 


Based  on  the  questionnaire  results,  participants  were  in  agreement  that  the  size  and  number  of 
swimlanes'  were  suitable  and  that  the  titles,  labels  and  colours  used  were  clear,  easy  to  read  and 
made  sense  with  the  exception  of  one  participant  who  felt  the  lettering  was  too  small.  Participants 
were  also  in  agreement  that  the  cursor  traveled  at  an  appropriate  speed  and  that  they  always  knew 
where  the  cursor  was.  One  participant  reported  that  at  times  it  was  difficult  to  find  the  cursor. 
Participants  agreed  that  it  was  easy  to  find  important  items  quickly  and  that  the  information  on 
the  screen  was  clearly  presented  and  was  easy  to  understand. 

The  questionnaire  results  for  the  usability  of  the  displayed  information  on  the  TE  functional 
display  are  illustrated  in  Figure  9. 
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Question 


QUESTION  LEGEND 


1 .  Size  of  threat  list  is  suitable. 

9. 1  always  know  where  the  cursor  is  on  the  TE  display. 

2.  Size  of  swimlanes  is  suitable. 

10.  The  cursor  always  travels  at  an  appropriate  speed. 

3.  Number  of  swimlanes  is  suitable. 

1 1 . 1  can  find  important  items  quickly. 

6.  Titles  and  labels  are  clear. 

16.  Organization  of  information  was  clear. 

7.  Colours  used  on  screen  make  sense  to  me. 

17.  Information  was  easy  to  understand. 

8.  Size  of  lettering  on  screen  is  easy  to  read. 

Figure  9:  TE  Questionnaire  Results  -  Display  of  Information 


*  Each  threat  that  appears  in  the  threat  list  on  the  TE  functional  display  will  be  allocated  to  a  “swimlane” 
that  spans  the  length  of  time  (or  range)  represented  by  the  x-axis.  Each  threat  (or  swimlane)  is  assigned  a 
relative  threat  ranking  number  with  the  highest  threat  ranked  ‘  1’  at  the  top  of  the  list. 
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5.3. 1.3  Usefulness  of  the  Information 


The  participants  positively  rated  ease  of  use  of  the  system,  their  comfort  level  when  using  the 
system,  their  understanding  of  what  the  system  was  doing  at  all  times  and  an  overall  satisfaction 
with  the  system.  Participants  agreed  that  it  was  easy  to  toggle  the  x-axis  scale  between  range  and 
time  and  easy  to  expand/reduce  the  x-axis.  Participants  also  agreed  that  they  did  not  need  to  learn 
a  lot  about  the  display  before  they  could  use  it. 

The  questionnaire  results  for  usefulness  of  information  on  the  TE  functional  display  are 
illustrated  in  Figure  10. 


QUESTION  LEGEND 


4. 1  found  it  easy  to  toggle  the  x-axis  scale  between  range  and 
time. 

5. 1  found  it  easy  to  expand/reduce  the  x-axis  scale. 

12. 1  found  the  threat  evaluation  functional  view  easy  to  use. 

13.  I  needed  to  learn  a  lot  about  the  threat  evaluation 
functional  view  before  I  could  use  it. 

14. 1  felt  comfortable  using  the  threat  evaluation  system. 


15.  The  threat  evaluation  functional  view  has  all  the 
functions  and  capabilities  I  expect  it  to  have. 

18.  I  knew  what  the  threat  evaluation  functional  view  system 
was  doing  at  all  times. 

19.  In  general,  the  system  provides  information  users  will 
use. 

20.  It  was  simple  to  use  this  system. 

21.  Overall,  I  am  satisfied  with  this  system. 


Figure  10:  TE  Questionnaire  Results  -  Usefulness  of  Information 


5.3.1. 4  Participant  Comments 

Participant  comments  were  recorded  during  the  conduct  of  the  trial.  Of  note,  participants: 

1 .  Stated  that  it  was  awkward  to  have  to  go  to  the  bottom  of  the  screen  to  expand/reduce  the  x- 
axis  scale.  It  was  suggested  that  the  operator  should  have  the  option  to  expand/reduce  the 
scale  at  either  the  top  or  the  bottom  of  the  screen. 
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2.  Suggested  that  operators  should  have  the  ability  to  increase  or  decrease  the  number  of 
swimlanes  displayed. 

3.  Noted  that  different  missile  types  should  be  represented  with  either  different  symbols  or 
different  colours. 

4.  Stated  that  the  threat  evaluation  functional  view  interface  was  too  cluttered.  It  was  suggested 
that  operators  should  have  the  option  of  selecting  what  information  they  would  like  to  see 
displayed. 

5.  Suggested  that  there  was  no  need  for  friendly  or  neutral  entities  to  be  displayed  in  the 
swimlanes. 

6.  Recommended  that  they  would  like  some  indication  as  to  where  a  threat  ‘jumped’  from  when 
a  category  jump  has  occurred.  It  was  suggested  that  the  swimlane  where  the  threat  came  from 
be  highlighted. 

7.  Preferred  the  time -based  x-axis  compared  to  the  range-based  x-axis. 

8.  Suggested  that  information  about  the  history  of  altitude  of  an  aircraft  would  be  useful. 

9.  Stated  that  they  would  prefer  a  3-D  threat  evaluation  functional  display  because  this  would 
allow  the  operators  to  see  altitude  aircraft  are  flying  at. 

10.  Suggested  that  each  operator  should  have  the  ability  to  tailor  where  information  on  the  threat 
evaluation  functional  display  is  displayed. 

11.  Stated  they  would  prefer  mechanical  functional  changes  (i.e.,  touch  screen)  to  interact  with 
the  information  on  the  display. 

12.  Indicated  that  the  capability  of  a  threat  was  more  important  than  the  hostile  intent  that  a  threat 
showed. 

13.  Suggested  that  they  would  prefer  to  use  a  joystick  over  a  mouse  to  move  the  cursor. 

14.  Indicated  that  they  would  prefer  to  have  the  threat  evaluation  and  the  combat  power 
management  display  combined  into  a  single  screen  or  presented  side  by  side.  Participants  did 
not  like  having  to  continually  look  from  side  to  side. 

15.  Stated  that  the  information  contained  in  the  pop-up  display  was  useful  but  it  was  difficult  to 
read  the  information  in  the  semi-transparent  pop-up  display  when  it  was  overlaying  another 
target. 

5.3.2  CPM  Functional  View 
5.3.2.1  General 

Questionnaire  rating  data  was  collated  and  summary  statistics  (mean  and  standard  deviation)  were 

calculated.  Mean  questionnaire  results  (N  =  6)  and  standard  deviations  for  the  Combat  Power 

Management  functional  display  system  are  shown  in  Figure  1 1  and  Figure  12.  The  following 

sections  highlight  the  participants’  feedback  as  gathered  through  the  questionnaires. 
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5.3.2.2  Display  of  Information 

Based  on  the  questionnaire  results,  participants  were  in  agreement  that  the  size  of  the  local 
engagement  plan  and  the  action  was  suitable  and  that  the  titles,  labels  and  colours  used  were 
clear,  easy  to  read  and  made  sense.  One  participant  felt  the  lettering,  the  local  engagement  plan 
and  the  actions  were  too  small.  Participants  were  also  in  agreement  that  the  cursor  travelled  at  an 
appropriate  speed  and  that  they  always  knew  where  the  cursor  was.  One  participant  reported  that 
at  times  it  was  difficult  to  find  the  cursor.  Participants  agreed  that  it  was  easy  to  find  important 
items  quickly  and  that  the  information  on  the  screen  was  clearly  presented  and  was  easy  to 
understand. 

The  questionnaire  results  for  the  usability  of  the  displayed  information  on  the  CPM  functional 
display  are  illustrated  in  Figure  1 1 . 
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QUESTION  LEGEND 

1.  Size  of  local  engagement  plan  is  suitable. 

2.  Size  of  actions  is  suitable. 

4.  Titles  and  labels  are  clear. 

5.  Colours  used  on  screen  make  sense  to  me. 

6.  Size  of  lettering  on  screen  is  easy  to  read. 


Question 


7. 1  always  know  where  the  cursor  is  on  the  CPM  display. 
8.  The  cursor  always  travels  at  an  appropriate  speed. 

9. 1  can  find  important  items  quickly. 

10.  CPM  display  is  easy  to  use. 

14.  Organization  of  information  was  clear. 

15.  Information  was  easy  to  understand. 


Figure  11:  CPM  Questionnaire  Results  -  Display  of  Information 


5.3.2.3  Usefulness  of  the  Information 

The  participants  positively  rated  ease  of  use  of  the  system,  their  comfort  level  when  using  the 
system,  their  understanding  of  what  the  system  was  doing  at  all  times  and  an  overall  satisfaction 
with  the  system.  Participants  agreed  that  it  was  easy  to  expand/reduce  the  x-axis.  Participants  also 
agreed  that  they  did  not  need  to  learn  a  lot  about  the  display  before  they  could  use  it. 


DRDC  Toronto  CR  2009-041 


The  questionnaire  results  for  the  usefulness  of  the  information  on  the  CPM  fiinetional  display  are 
illustrated  in  Figure  12. 


QUESTION  LEGEND 


3.  I  found  it  easy  to  expand/reduce  the  x-axis  scale. 

11.  I  needed  to  leam  a  lot  about  the  Combat  Power 
Functional  View  before  I  could  use  it. 

12. 1  felt  comfortable  using  the  Combat  Power  system. 

13.  The  Combat  Power  Functional  View  has  all  the  functions 
and  capabilities  I  expect  it  to  have. 


16.  I  knew  what  the  Combat  Power  Functional  View  system 
was  doing  at  all  times.  17.  In  general,  the  system  provides 
information  users  will  use. 

18.  It  was  simple  to  use  this  system. 

19.  Overall,  I  am  satisfied  with  this  system. 


Figure  12:  CPM  Questionnaire  Results  -  Usefulness  of  Information 


5. 3.2.4  Participant  Comments 

Participant  comments  were  recorded  during  the  conduct  of  the  trial.  Of  note,  participants: 

1 .  Liked  the  time-line  presented  on  the  combat  power  management  engagement  plan. 

2.  Liked  the  engagement  plans  comprising  of  actions  that  were  provided. 

3.  Stated  that  they  would  like  the  ability  to  veto  all  actions  and  to  control  what  actions  will  be 
executed. 

4.  Suggested  that  they  would  like  to  see  the  engagement  plans  for  simultaneous  missiles 
presented  on  the  same  lines  (as  opposed  to  having  engagement  plans  stacked). 

5.  Stated  that  the  combat  power  management  display  interface  was  too  busy. 

6.  Expressed  concerns  that  operators  may  not  trust  the  engagement  plans  provided  by  the 
automation  component  of  INCOMMANDS  CDSC. 
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5.4  Utility  Questionnaire  Results 

5.4.1  TE  Functional  View 

5.4. 1.1  General 

Questionnaire  rating  data  was  collated  and  summary  statistics  (mean  and  standard  deviation)  were 
calculated.  Mean  questionnaire  results  (N  =  6)  and  standard  deviations  (eyelashes)  for  the  Threat 
Evaluation  functional  display  system  are  in  Figure  13  and  Figure  14.  The  following  sections 
highlight  the  participants’  feedback  as  gathered  through  the  questionnaires. 

5.4. 1.2  Overall  Assessment  of  Threats 

Overall,  the  participant  ratings  on  the  utility  of  the  threat  evaluation  functional  display  were  very 
favourable.  All  participants  agreed  that  the  threat  evaluation  functional  display  improved 
operators’  ability  to  make  decisions  with  respect  to  goals  and  processes  pertaining  to  the 
assessment  of  threats  in  the  environment.  Additionally,  all  participants  agreed  that  the  threat 
evaluation  view  always  made  it  clear  where  the  threats  were  in  relationship  to  the  ownship  and  it 
was  always  evident  when  a  threat  made  a  category  jump  (e.g.,  from  mid-level  to  a  high-level). 
However,  some  participants  felt  that  the  threat  evaluation  view  did  not  allow  operators  to  form  an 
accurate  mental  picture  of  their  environment  and  did  not  allow  operators  to  easily  view  all  of  the 
information  required  to  evaluate  all  the  threats.  One  participant  reported  that  the  TE  functional 
view  did  not  provide  operators  with  a  more  comprehensive  assessment  of  all  threats,  did  not 
prevent  operators  from  “tunnelling”  on  the  highest  priority  threat,  and  did  not  allow  operators  to 
quickly  and  easily  evaluate  all  the  threats  in  the  environment. 

The  questionnaire  results  for  overall  assessment  of  threats  using  the  threat  evaluation  view  are 
presented  in  Figure  13. 
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Question 


QUESTION  LEGEND 


1.  Overall,  the  use  of  the  Threat  Evaluation  View  improved 
my  ability  to  make  decisions  with  respect  to  the  goals  and 
processes  pertaining  to  the  assessment  of  threat  in  the 
environment. 

2.  The  Threat  Evaluation  View  provided  me  with  a  more 
comprehensive  assessment  of  all  threats  in  the  environment. 

3.  The  Threat  Evaluation  View  allowed  me  to  form  an 
accurate  mental  picture  of  my  environment. 

4.  The  Threat  Evaluation  View  always  made  it  clear  where 
the  targets  were  in  relationship  to  the  ownship. 

5.  It  was  always  clear  when  a  threat  made  a  category  jump 
(e.g.,  from  mid-level  threat  to  high-level  threat). 

6.  The  Threat  Evaluation  View  helped  me  to  avoid 
“tunnelling”  on  the  highest  priority  threat. 


7.  It  was  useful  having  the  ability  to  switch  between  a  time 
and  a  range  scale  (nautical  miles)  on  the  Threat  Evaluation 
View. 

8.  The  information  in  the  roll-over  menu  helped  me  to 
quickly  gather  information  about  the  threats  in  the 
environment. 

9.  The  interface  on  the  Threat  Evaluation  View  allowed  me 
to  easily  view  all  of  the  information  required  to  evaluate  all 
the  threats  in  my  environment. 

10.  I  found  that  the  location  of  all  the  information  on  the 
Threat  Evaluation  View  interface  allowed  me  to  quickly  and 
easily  evaluate  all  the  threats  in  my  environment. 

18.  Overall,  the  Threat  Evaluation  View  is  a  valuable  tool  for 
evaluating  threats  in  the  environment. 


Figure  13:  TE  Questionnaire  Results  -  Overall  Assessment  of  Threats 


Participant  comments  were  recorded  during  the  conduct  of  the  trial  as  it  pertained  to  the  utility  of 

the  TE  functional  view  for  assessing  threats  in  the  environment.  Of  note,  participants: 

1 .  Stated  that  the  threat  evaluation  view  would  be  most  important  in  a  littoral  setting. 

2.  Noted  that  the  threat  evaluation  view  was  extremely  useful  for  assessing  threats  in  the 
environment.  Participants  raised  concern  over  the  potential  negative  impact  of  providing  the 
operator  with  too  much  information. 

3.  Expressed  concern  about  how  they  would  develop  a  proper  level  of  trust  in  the  system  and 
avoid  either  mistrust  or  distrust. 

4.  Stated  that  the  threat  evaluation  view  prevented  operators  from  “tunnelling  in”  on  the  threat 
closest  in  proximity  to  the  ship.  In  many  instances,  the  highest  threat  is  assumed  to  be  the 
contact  closest  to  the  ship  which  may  not  necessarily  be  true. 
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5.  Suggested  that  having  the  threat  explicitly  identified  on  the  threat  evaluation  view  (e.g.,  Mig 
20)  would  be  useful. 

6.  Noted  that  the  arrow  that  appears  when  a  threat  makes  a  category  jump  is  too  small. 

7.  Stated  that  because  capability  is  more  important  than  intent,  they  suggested  that  the  order  in 
the  drill-down  pane  be  flipped  to  reflect  this. 

8.  Suggested  that  having  course  “be  true”  (e.g.,  course  heading  of  45°  instead  of  -15°)  would  be 
more  useful. 

5.4. 1.3  Impact  on  Workload,  Situation  Awareness  and  Operations  Room 
Communications 

Participants  reported  that  their  overall  workload  decreased  when  they  were  using  the  threat 
evaluation  view  compared  to  using  only  the  physical  display.  They  also  reported  that  overall 
workload  would  decrease  in  a  high  workload  situation  relative  to  a  low  workload  situation. 
Participants  also  indicated  that  their  overall  situation  awareness  was  better  when  using  the  threat 
evaluation  view  compared  to  using  only  the  physical  display.  Participants  also  suggested  that 
their  overall  situation  awareness  would  increase  in  a  high  workload  situation  compared  to  a  low 
workload  situation.  Participants  also  stated  that  they  would  feel  comfortable  using  the  threat 
evaluation  view  under  time  pressure.  Finally,  participants  reported  that  the  presence  of  the  threat 
evaluation  view  would  not  interfere  with  the  flow  of  communications  in  the  operations  room  and 
that  the  threat  evaluation  view  would  make  the  operator’s  job  easier. 

The  questionnaire  results  for  workload,  situation  awareness  and  operations  room  communications 
when  using  the  threat  evaluation  functional  display  can  be  found  in  Figure  14. 
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QUESTION  LEGEND 

12.  The  threat  evaluation  view  decreased  my  overall  16.  The  presence  of  the  threat  evaluation  view  will  not 

workload.  interfere  with  the  flow  of  communication  in  the  operations 

room. 

13.  The  threat  evaluation  view  would  decrease  my  overall 

workload  more  in  a  high  workload  situation  relative  to  a  low  17.  I  would  feel  comfortable  using  the  threat  evaluation  view 
workload  situation.  under  time  pressure. 

14.  The  threat  evaluation  view  increased  my  overall  situation  19.  Overall,  the  threat  evaluation  view  will  make  the 

awareness.  operator’s  job  easier. 

15.  The  threat  evaluation  view  would  increase  my  overall 
situation  awareness  more  in  a  high  workload  situation 
relative  to  a  low  workload  situation. 

Figure  14:  TE  Questionnaire  Results  -  Impact  on  Workload,  SA,  &  Ops  Room 

Communications 


Participant  comments  were  recorded  during  the  conduct  of  the  trial  as  it  pertained  to  the  impact  of 
the  TE  functional  display  on  workload,  situation  awareness,  and  operations  room 
communications.  Of  note,  participants: 

1.  Reported  that  the  physical  display  is  needed  in  conjunction  with  the  threat  evaluation  view  in 
order  to  provide  the  operator  with  good  situation  awareness.  They  suggested  that  the  physical 
display  helps  provide  an  “overall  picture”  of  the  environment  and  that  the  threat  evaluation 
view  provides  more  information  about  specific  threats.  The  physical  display  provides 
information  about  friendly  units,  consorts,  etc.  whereas;  the  threat  evaluation  view  does  not 
provide  this  information. 

2.  Stated  that  the  threat  evaluation  view  would  actually  decrease  the  need  for  communication  in 
the  operations  room. 

3.  Suggested  that  having  more  than  one  pop-up  display  would  allow  the  operator  to  compare 
information  across  tracks  and  this  would  likely  lead  to  better  situation  awareness. 

4.  Noted  seeing  the  threats  ranked  on  the  threat  evaluation  view  lowered  their  workload. 
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5.  Suggested  that  having  the  threat  evaluation  view  would  not  necessarily  reduce  attentional 
demands  but  it  may  free  up  attentional  demands  that  can  be  allocated  to  other  tasks. 

5.4.2  CPM  Functional  View 

5.4.2. 1  General 

Questionnaire  rating  data  was  collated  and  summary  statistics  (mean  and  standard  deviation)  were 
calculated.  Mean  questionnaire  results  (N  =  6)  and  standard  deviations  (eyelashes)  for  the  CPM 
functional  display  system  are  in  Figure  15  and  Figure  16.  The  following  sections  highlight  the 
participants’  feedback  as  gathered  through  the  questionnaires. 

5.4.2.2  Overall  Assessment  of  Engagement  Plans 

Participants’  ratings  for  the  engagement  plans  presented  on  the  combat  power  management 
display  were  very  favourable.  Participants  reported  that  the  combat  power  management  view 
improved  their  ability  to  make  decisions  with  respect  to  management  and  application  of  combat 
power  and  improved  their  ability  to  generate  an  ‘optimal’  global  engagement  plan.  Participants 
said  that  the  interface  would  allow  them  to  easily  view  all  of  the  information  required  to  evaluate 
the  presented  engagement  plan  and  to  determine  if  the  plan  would  be  successful.  Further, 
participants  stated  that  the  location  of  the  information  on  the  combat  power  management  display 
allowed  them  to  quickly  and  easily  evaluate  the  engagement  plan.  Finally,  two  participants 
reported  that  the  combat  power  management  display  would  not  be  helpful  for  assessing  the 
outcome  of  an  engagement. 

The  questionnaire  results  of  engagement  plans  presented  on  the  combat  power  management  view 
can  be  seen  in  Figure  15. 
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20  21  22  23  24  25  26 

Question 


QUESTION  LEGEND 


20.  Overall,  the  Combat  Power  Functional  View  would 
improve  my  ability  to  make  decisions  with  respect  to  the 
management  of  combat  power. 

21.  Overall,  the  use  of  the  Combat  Power  Functional  View 
would  improve  my  ability  to  make  decisions  with  respect  to 
the  application  of  combat  power. 

22.  The  Combat  Power  Functional  View  would  improve  my 
ability  to  generate  an  ‘optimum’  global  engagement  plan. 

23.  During  the  execution  of  a  combat  power  plan,  the 
Combat  Power  Functional  View  would  help  me  to  assess  the 
outcome  of  an  engagement. 


24.  The  interface  on  the  Combat  Power  Functional  View 
allowed  me  to  easily  view  all  of  the  information  required  to 
evaluate  the  engagement  plan  presented  on  the  display. 

25.  The  interface  on  the  Combat  Power  Functional  View 

allowed  me  to  easily  view  all  of  the  information 

required  to  evaluate  whether  the  presented  engagement  plan 
would  be  a  successful  plan. 

26.  I  found  the  location  of  all  the  information  on  the  Combat 
Power  Functional  View  interface  allowed  me  to  quickly  and 
easily  evaluate  the  engagement  plan  presented  on  the  display. 


Figure  15:  CPM  Questionnaire  Results  -  Overall  Assessment  of  Engagement  Plans 


Participant  comments  were  recorded  during  the  conduct  of  the  trial  as  it  pertained  to  an  overall 

assessment  of  the  engagement  plan.  Of  note: 

1.  A  single  participant  suggested  incorporating  the  combat  power  management  and  the  threat 
evaluation  displays  into  one  display  or  have  the  two  monitors  adjacent  to  one  another. 

2.  A  single  participant  stated  that  the  font  used  on  the  drill-down  menu  on  the  right-hand  side  of 
the  monitor  was  too  small  and  this  made  it  very  difficult  to  read. 

3.  Participants  suggested  that  a  visual  prompt  warning  the  operators  when  an  engagement  will 
occur  would  be  useful  (e.g.,  launch  chaff  in  ten  seconds). 

4.  Participants  said  they  would  like  to  have  evasive  manoeuvres  (e.g.,  moving  the  ship  to 
another  location)  as  one  of  the  options  on  the  combat  power  management  view. 

5.  A  single  participant  felt  the  default  time  line  on  the  combat  power  management  view  was  too 
small  and  that  the  plans  were  only  visible  for  a  short  period  of  time. 
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5.4.2.3  Impact  on  Workload,  Situation  Awareness  and  Operations  Room 
Communications. 

Participants  agreed  that  their  overall  workload  decreased  when  they  were  using  the  combat 
management  view  compared  to  using  only  the  physical  display.  They  also  reported  that  overall 
workload  would  decrease  in  a  high  workload  situation  relative  to  a  low  workload  situation. 
Participants  also  indicated  that  their  overall  situation  awareness  was  better  when  using  the  combat 
power  management  view  compared  to  using  only  the  physical  display.  However,  one  participant 
reported  that  the  combat  power  functional  view  would  not  increase  his  situation  awareness 
because  he  would  use  the  physical  display  for  his  situation  awareness.  Participants  also 
suggested  that  their  overall  situation  awareness  would  increase  in  a  high  workload  situation 
compared  to  a  low  workload  situation.  Participants  also  stated  that  they  would  feel  comfortable 
using  the  combat  power  management  view  under  time  pressure.  Finally,  participants  reported  that 
the  presence  of  the  combat  power  management  view  would  not  interfere  with  the  flow  of 
communications  in  the  operations  room  and  that  the  combat  power  management  view  would 
make  the  operator’s  job  easier. 

The  questionnaire  results  for  workload,  situation  awareness  and  operations  room  communications 
when  using  the  combat  power  management  functional  display  can  be  found  in  Figure  16. 


QUESTION  LEGEND 


28.  The  Combat  Power  Functional  View  would  decrease  my 
overall  workload. 

29.  The  Combat  Power  Functional  View  would  decrease  my 
overall  workload  more  in  a  high  workload  situation  relative 
to  a  low  workload  situation. 

30.  The  Combat  Power  Functional  View  would  increase  my 
overall  situation  awareness. 

31.  The  Combat  Power  Functional  View  would  increase  my 
overall  situation  awareness  more  in  the  high  workload 
situation  relative  to  the  low  workload  situation. 


32.  The  presence  of  the  Combat  Power  Functional  View 
would  not  interfere  with  the  flow  of  communication  in  the 
operations  room. 

33.  I  would  feel  comfortable  using  the  Combat  Power 
Functional  View  under  time  pressure. 

34.  Overall,  the  Combat  Power  Functional  View  is  a  valuable 
tool  for  managing  and  applying  combat  power. 


Figure  16:  CPM  Questionnaire  Results  -  Impact  on  Workload,  SA  &  Ops  Room 

Communications 


DRDC  Toronto  CR  2009-041 


Participant  comments  were  recorded  during  the  conduct  of  the  trial  as  it  pertained  to  the  impact  of 
the  CPM  functional  display  on  workload,  situation  awareness,  and  operations  room 
communications.  Of  note,  participants: 

1.  Stated  that  the  combat  power  management  view  in  conjunction  with  both  the  physical  and 
threat  evaluation  display  would  improve  overall  situation  awareness. 

2.  Reported  that  the  combat  power  management  view  would  actually  decrease  the  need  for 
communications  in  the  operations  room. 

3.  Reported  that  having  the  combat  power  management  view  would  reduce  the  amount  of 
attentional  resources  required  for  combat  power  management  and  in  turn,  this  would  allow 
operators  to  direct  their  attention  to  other  operational  tasks. 

4.  Indicated  that  the  combat  power  management  view  would  be  most  useful  for  multiple 
engagements. 

5.4.3  Overall  TE  and  CPM  View 

5.4.3.1  Impact  on  Workload  and  Situation  Awareness 

Participants  reported  that  together,  the  threat  evaluation  and  combat  power  management  views 
would  decrease  overall  workload  and  increase  overall  situation  awareness.  The  questionnaire 
results  for  overall  workload  and  situation  awareness  for  the  threat  evaluation  and  combat  power 
management  view  can  be  found  in  Figure  17. 
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QUESTION  LEGEND 

35.  Overall,  both  the  Threat  Evaluation  View  and  the  Combat  36.  Overall,  both  the  Threat  Evaluation  View  and  the  Combat 
Power  Functional  View  would  decrease  my  workload.  Power  Functional  View  would  increase  my  situation 

awareness. 

Figure  1 7:  TE  &  CPM  Questionnaire  Results  -  Impact  on  Workload  and  SA 


Participant  comments  were  recorded  during  the  conduct  of  the  trial  as  it  pertained  to  the  impact  of 
both  the  TE  and  CPM  functional  displays  on  workload  and  situation  awareness.  Of  note, 
participants: 

1.  Agreed  that  the  INCOMMANDS  CDSC  is  a  much  better  system  than  the  one  currently  being 
used. 

2.  Stated  that  the  INCOMMANDS  CDSC  would  greatly  reduce  workload  and  increase  overall 
situation  awareness. 

5.4.4  Summary 

Overall,  the  feedback  gathered  was  very  positive  and  further  supports  that  the  proposed 
INCOMMANDS  OMI  concepts  are  in  line  with  the  current  operational  requirements  of  the  ORO 
and  SWC  Naval  operators.  To  that  end,  the  observations  can  be  summarized  as  follows: 

1.  The  participants  positively  rated  ease  of  use  of  the  system,  their  comfort  level  when  using  the 
system,  their  understanding  of  what  the  system  was  doing  at  all  times  and  an  overall 
satisfaction  with  the  system. 

2.  Utility  ratings  of  both  the  TE  and  CPM  functional  views  were  favourable. 

3.  Participants  reported  that  their  overall  workload  decreased  when  they  were  using  the 
INCOMMANDS  CDSC  compared  to  using  only  the  physical  display.  They  also  reported  that 
overall  workload  would  decrease  in  a  high  workload  situation  relative  to  a  low  workload 
situation. 
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4.  Participants  indicated  that  their  overall  situation  awareness  was  better  when  using  the 
INCOMMANDS  CDSC  compared  to  using  only  the  physical  display.  Participants  also 
suggested  that  their  overall  situation  awareness  would  increase  in  a  high  workload  situation 
compared  to  a  low  workload  situation. 

5.  Participants  stated  that  they  would  feel  comfortable  using  the  INCOMMANDS  CDSC  under 
time  pressure. 

6.  Participants  reported  that  the  presence  of  the  INCOMMANDS  CDSC  would  not  interfere 
with  the  flow  of  communications  in  the  operations  room  and  that  the  threat  evaluation  view 
would  make  the  operator’s  job  easier. 

Most  participants  felt  that  they  would  prefer  to  have  spent  more  time  with  the  INCOMMANDS 
CDSC  in  order  to  get  a  better  appreciation  of  how  they  would  use  the  system  in  a  littoral  setting. 
Participants  said  they  liked  how  the  threat  evaluation  and  combat  power  management  applications 
were  presented  separately  because  this  allowed  them  to  have  a  better  overall  mental  picture  of 
events  occurring  in  the  environment  and  how  they  might  respond  to  a  target.  Even  though 
responses  were  positive,  participants  did  have  recommendations  for  how  the  system  could  be 
improved.  For  example,  participants  wanted  to  know  explicitly  what  the  threat  was  (e.g.,  Mig  20) 
and  they  wanted  to  have  more  freedom  to  tailor  the  displays  to  suit  individual  operator’s  needs. 
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6  Conclusion  Material 


6.1  General 

The  primary  objective  of  this  evaluation  effort  was  to  both  identify  usability  problems  with  and 
determine  the  utility  of  the  INCOMMANDS  CDSC.  The  methodology  employed  two 
complementary  techniques:  Heuristic  Evaluation  and  Usability  and  Utility  Testing.  To  that  end, 
the  evaluations  collectively: 

1 .  Identified  several  usability  problems  related  to  the  set  of  established  OMl  guidelines  which  in 
turn  provide  distinct  focal  points  for  the  development  of  future  versions  of  the 
INCOMMANDS  CDSC;  and 

2.  Collected  operator  feedback  regarding  the  usability  and  utility  of  the  INCOMMANDS 
System  as  it  pertains  to  the  performance  of  TE  and  CPM  in  the  ‘real  world’; 

Furthermore,  the  Usability  and  Utility  Testing  provided  a  significant  series  of  lessons  learned  that 
would  provide  useful  direction  for  future  experiments. 

6.2  Conclusions 

Based  on  the  evaluation  of  the  INCOMMANDS  CDSC,  it  is  concluded  that: 

1.  Both  evaluation  techniques  provide  complementary  insight  into  improving  the 
INCOMMANDS  CDSC.  The  Heuristic  Evaluation  helps  to  ensure  that  the  design  of  system 
is  compliant  with  HCl  best  practices.  Whereas  the  Usability  and  Utility  Testing  with  Naval 
operators  ensures  that  contextual  feedback  is  obtained  which  is  lacking  from  the  former 
evaluation  technique. 

2.  The  results  from  the  Heuristic  Evaluation  provide  specific  direction  to  support  future 
development  of  the  INCOMMANDS  CDSC  in  order  to  improve  the  overall  usability  of  the 
system.  As  such,  any  additional  work  to  be  completed  on  the  system  should  attempt  to 
improve  the  usability  of  the  system  by  both  fixing  the  identified  non-compliances  as  well  as 
adding  the  missing  functionality. 

3.  Based  on  the  participant  feedback  received  during  the  Usability  and  Utility  Testing,  the 
following  preliminary  conclusions  can  be  postulated  (additional  studies  are  recommended  to 
further  validate  these  claims): 

a.  The  proposed  INCOMMANDS  CDSC  OMI  concepts  are  in  line  with  the  current 
operational  requirements  of  the  ORO  and  SWC  Naval  operators  and  would  make 
their  job  easier. 

b.  The  participants  positively  rated  ease  of  use  of  the  system,  their  comfort  level  when 
using  the  system,  their  understanding  of  what  the  system  was  doing  at  all  times  and 
an  overall  satisfaction  with  the  system. 

c.  Overall  workload  would  decrease  when  using  the  INCOMMANDS  CDSC  compared 
to  using  only  the  physical  display  for  TE  and  CPM  activities.  Furthermore,  overall 


DRDC  Toronto  CR  2009-041 


workload  would  decrease  in  a  high  workload  situation  relative  to  a  low  workload 
situation. 

d.  Overall  situation  awareness  would  improve  when  using  the  INCOMMANDS  CDSC 
compared  to  using  only  the  physical  display.  Overall  situation  awareness  would 
increase  in  a  high  workload  situation  compared  to  a  low  workload  situation. 

e.  The  INCOMMANDS  CDSC  can  be  employed  in  situations  subject  to  time  pressure. 

f.  The  INCOMMANDS  CDSC  would  not  interfere  with  the  flow  of  communications  in 
the  operations  room. 

6.3  Recommendations 

Stemming  from  the  HF  evaluation  of  the  INCOMMANDS  CDSC,  the  following 

recommendations  are  put  forth: 

1.  Improve  the  INCOMMANDS  CDSC  based  on  the  direction  summarized  in  this  report;  and 

2.  Conduct  additional  evaluations  with  other  techniques  to  both  verify  incremental 
improvements  to  the  OMl  design  as  well  as  substantiate  initial  conclusions  regarding  the 
potential  benefits  afforded  by  the  INCOMMANDS  CDSC  as  stated  herein.  The  proposed 
experimental  plan  in  Annex  F  is  a  good  starting  point  for  further  evaluations. 
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Annex  A  Heuristic  Evaluation  Compliance  Matrix 


A.1  General 

The  results  from  the  Heuristic  Evaluation  of  the  INCOMMANDS  CDSC  are  summarized  in  the 

compliance  matrix  in  this  Annex.  The  column  headings  for  the  compliance  matrix  are  the 

following: 

1.  Compliance.  The  Compliance  column  places  each  design  feature  in  one  of  three  categories 
as  follows: 

a.  Not  compliant.  The  INCOMMANDS  CDSC  does  not  satisfy  the  objectives  of  the 
stated  guideline; 

b.  Out  of  Scope  -  Experimental.  Compliance  to  the  guideline  could  not  be  evaluated 
via  inspection.  It  was  anticipated  that  the  guideline  could  be  verified  through 
usability  testing  or  experimentation;  and 

c.  Out  of  Scope.  Design  features  that  are  out  of  the  current  INCOMMANDS  CDSC 
build.  Any  instances  in  which  the  Scientific  Authority  and  HE  analyst  agree  that  an 
element  identified  in  the  compliance  matrix  is  out  of  the  scope  of  the  current 
INCOMMANDS  CDSC,  that  design  element  should  be  identified  and  reserved  for 
consideration  in  future  development  of  the  INCOMMANDS  CDSC 

2.  Paragraph.  Column  two  identifies  the  paragraph  number  of  the  guideline  INCOMMANDS 
Human  Factors  Design  and  Evaluation  Guide  (Version  1.0)  that  is  relevant  to  the  design 
feature. 

3.  INCOMMANDS  HF  Design  and  Evaluation  Guide.  Column  3  contains  the  text  of  the 
guideline  that  is  relevant  to  the  design  feature. 

4.  INCOMMANDS  CDSC.  Column  4  describes  each  identified  design  feature  of  the 
INCOMMANDS  CDSC  OMl. 

5.  Severity  Rating.  Column  5  describes  whether  the  severity  of  the  non-compliance.  A  minor 
non-compliance  is  deemed  to  have  minimal  impact  on  the  ability  of  the  operator  to  complete 
the  task  at  hand.  A  major  non-compliance  is  considered  to  severely  impact  the  ability  of  the 
operator  to  complete  the  task  at  hand. 
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Compliance 

Paragraph 

INCOMMANDS  HE  Design  and  Evaluation 
Guide 

INCOMMANDS  CDSC 

Severity 

Rating 

7  Electronic  Support  System  (ESS)  OMI  DESIGN  GUIDELINES 

7.1  GENERAL  DESIGN  GOALS 

7,1,1  EMPLOY  OPERATOR  CENTERED  PRINCIPLES 

Experimental  -  Out  of 
Scope 

7.1. 1.3 

The  ESS  shall  help  or  enable  the  operators  to  carry 
out  their  responsibilities  and  tasks  safely, 
efficiently,  and  effectively.  [Carrying  out  a  task 
effectively  means  producing  the  desired  result. 
Carrying  out  a  task  efficiently  means  that  the 
desired  result  is  produced  with  a  minimum  of  waste 
(usually  in  relation  to  time)]. 

These  measures  would  have  to  be  collected 
and  analyzed  during  the  experimental 
phase. 

TBD 

Experimental  -  Out 
of  scope 

7.1. 1.4 

The  operator  shall  always  have  final  authority  over 
the  allocation  of  ESS  functions  (i.e.,  task  allocated 
to  human  and/or  system). 

The  current  INCOMMANDS  CDSC  does 
not  support  final  authority  over  functions 
to  the  operator.  A  future  version  is 
intended  to  allow  operators  to  have  final 
authority  over  all  INCOMMANDS  CDSC 
functions. 

Minor 

Experimental  -  Out  of 
Scope 

7.1. 1.5 

Functions  shall  be  automated  only  to  attain  greater 
overall  effectiveness,  efficiency,  reliability, 
simplicity,  economy,  and  system  safety  without 
reducing  human  involvement,  situation  awareness, 
or  human  performance  in  carrying  out  the  intended 
task. 

Performance  measures  and  situation 
awareness  measures  would  have  to  be 
collected  and  analyzed  during  the 
experimental  phase. 

TBD 

7,1,2  OPTIMIZE  HUMAN-SYSTEM  INTERACTION 

Not  Compliant 

7.1.2.1 

Inform  the  operator  of  system  failure  or 
degradation;  inform  the  operator  if  potentially 
unsafe  modes  are  manually  selected;  do  not 
interfere  with  manual  task  performance;  and  allow 
for  manual  override. 

The  current  INCOMMANDS  CDSC  does 
not  inform  the  operator  of  system  failure, 
or  degradation  or  if  any  unsafe  modes  are 
manually  selected.  The  system  does  not 
allow  for  manual  override. 

Major 
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Compliance 

Paragraph 

INCOMMANDS  HE  Design  and  Evaluation 
Guide 

INCOMMANDS  CDSC 

Severity 

Rating 

Experimental  -  Out  of 
Scope 

7.1.2.5 

ESS  functioning  shall  not  increase  the  demands  for 
cognitive  resources  (thinking  or  conscious  mental 
processes). 

The  NASA-  TEX,  and  usability  and  utility 
questionnaires  would  need  to  be  used  to 
assess  operator  workload. 

TBD 

Experimental  -  Out  of 
Scope 

7.1.2.6 

Extreme  levels  of  workload  (low  or  high)  due  to 

ESS  functioning  shall  be  avoided  (to  maximize 
operator-in-the-loop  and  reduce  automation  bias). 

The  NASA-TEX  and  performance 
measures  would  need  to  be  used  to  assess 
workload  and  to  ensure  maximum 
operator-in-the-loop. 

TBD 

Experimental  -  Out  of 
Scope 

7.1.2.7 

Operator  interaction  with  the  ESS  shall  not  require 
the  operator  to  take  significant  amounts  of  attention 
away  from  the  primary  task. 

The  performance  measures  would  need  to 
be  used  to  assess  if  the  ESS  requires 
operators  to  take  significant  amounts  of 
attention  away  from  the  primary  task. 

TBD 

Experimental  -  Out  of 
Scope 

7.1.2.8 

ESS  shall  not  interrupt  at  inappropriate  times  such 
as  during  periods  of  high  workload  or  during 
critical  moments  in  a  process. 

The  performance  measures  would  need  to 
be  used  to  assess  if  the  ESS  interrupts 
operators  during  periods  of  high  workload 
or  critical  moments. 

TBD 

Experimental  -  Out  of 
Scope 

7.1.2.9 

An  ESS  task  shall  be  less  difficult  to  carry  out  than 
the  manual  task  it  replaces. 

The  performance  measures  would  need  to 
be  used  and  the  usability  and  utility 
questionnaires  were  used  to  assess  task 
difficulty  when  using  the  ESS. 

TBD 

Experimental  -  Out  of 
Scope 

7.1.2.10 

Data  that  are  needed  by  the  operator  shall  be  easily 
accessible. 

The  performance  measures  would  need  to 
be  used  and  the  usability  and  utility 
questionnaires  were  used  to  assess  if  data 
are  easily  accessible. 

TBD 

Experimental  -  Out  of 
Scope 

7.1.2.11 

The  ESS  shall  allow  the  operator  to  interact  directly 
with  objects  which  are  important  to  the  operator’s 
tasks 

The  performance  measures  would  need  to 
be  used  and  the  usability  and  utility 
questionnaires  were  used  to  assess  if 
operators  are  able  to  interact  directly  with 
objects  important  to  operator’s  task. 

TBD 

7.1,3  PROMOTE  ESS  ROBUSTNESS  AND  RESILIENCE  TO  OPERATOR  ERROR 
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Compliance 

Paragraph 

INCOMMANDS  HE  Design  and  Evaluation 
Guide 

INCOMMANDS  CDSC 

Severity 

Rating 

Experimental  -  Out  of 
Scope 

7.1.3. 1  - 
7.1.3.14 

These  items  focus  on  ESS  robustness  and  resilience 
to  operator  error. 

Performance  measures,  trust,  workload, 
and  situation  awareness  questionnaires 
would  need  to  be  used  and  usability  and 
utility  questionnaires  were  used  to  assess 
these  items. 

TBD 

7.1,3  SUPPORT  OPE] 

RATOR  MONITORING  OF  ESS  FUNCTIONING 

Not  Compliant 

7.1.4.1 

Informative  feedback  shall  be  given  in  case  of  an 
ESS  failure;  such  as  the  likely  cause  and/or  location 
of  the  failure 

The  current  INCOMMANDS  CDSC  does 
not  provide  feedback  in  case  of  an  ESS 
failure. 

Major 

Not  Compliant 

7. 1.4.2 

The  ESS  shall  be  designed  so  that  operators  are 
able  to  monitor  the  ESS  and  the  functionality  of  its 
hardware  and  software,  including  the  display  of 
status  and  trend  information,  as  needed 

The  current  INCOMMANDS  CDSC  does 
not  allow  operators  to  monitor  the  ESS  and 
the  functionality  of  its  hardware  and 
software. 

Major 

Experimental  -  Out  of 
Scope 

7. 1.4.4 

This  item  deals  with  operator  cognitive  load  issues 
when  monitoring  the  ESS. 

Workload  measures  would  need  to  be  used 
to  assess  this  item. 

TBD 

Not  Compliant 

7.1.4.7 

The  ESS  shall  provide  some  type  of  indication  the 
system  is  still  being  monitored  by  some  automatic 
system. 

The  current  INCOMMANDS  CDSC  does 
not  provide  indication  that  the  system  is 
being  monitored  by  some  automatic 
system. 

Minor 

Not  Compliant 

7.1.4.8 

Critical  ESS  functions  shall  be  independently 
monitored  by  the  operator.  A  critical  function  is  a 
function  that  can  cause  system  failure  when  a 
malfunction  is  not  attended  to  immediately. 

The  current  INCOMMANDS  CDSC  does 
not  provide  system  failure  notices. 

Major 

Not  Compliant 

7.1.4.13 

The  ESS  shall  provide  means  to  indicate  to  the 
operator  that  data  are  missing,  incomplete, 
unreliable,  or  invalid  or  that  the  system  is  relying 
on  backup  data. 

The  current  INCOMMANDS  CDSC  does 
not  show  data  that  are  missing,  incomplete, 
unreliable  or  invalid  or  if  the  system  is 
relying  on  backup  data. 

Major 

7.2  EMPLOY  OPERATOR-CENTERED  OMI  DESIGN 

Experimental  -  Out  of 

7.2.1  -7.2.2 

These  two  items  assess  if  the  information  displays 

Questionnaires  and  usability  and  utility 

TBD 
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Compliance 

Paragraph 

INCOMMANDS  HF  Design  and  Evaluation 
Guide 

INCOMMANDS  CDSC 

Severity 

Rating 

Scope 

are  intuitive,  easy  to  understand  and  easy  to  use  and 
if  the  ESS  is  simple  for  operators  to  learn. 

questionnaires  were  used  to  assess  these 
two  items. 

7.3  SUPPORT  DIFFE 

RENT  MODES  OF  OPERATION 

Out  of  Scope 

7.3.3 

Seldom-used  ESS  modes  and  functions  shall  be 
clearly  identified.  As  ESSs  become  more  complex 
with  many  modes  and  functions,  the  cognitive 
burden  caused  by  the  need  for  mode  awareness 
increases.  Seldom-used  ESS  modes  and  functions 
will  pose  the  largest  burden  on  the  operator  because 
of  a  lack  of  familiarity.  Enabling  the  operator  to 
immediately  recognize  the  purpose  of  ESS  modes 
and  functions  can  lessen  this  burden. 

The  current  INCOMMANDS  CDSC  does 
not  have  seldom-used  modes.  However,  as 
the  system  evolves  seldom-used  modes 
and  function  will  need  to  be  clearly 
identified. 

Minor 

Out  of  Scope 

7.3.7 

The  ESS  shall  alert  the  operator  to  the  implications 
of  interactions  between  modes,  especially  when 
they  are  potentially  hazardous. 

The  current  INCOMMANDS  CDSC  does 
not  alert  the  operator  to  the  implications  of 
interactions  between  modes. 

Major 

Out  of  Scope 

7.3.8 

The  ESS  shall  either  prevent  the  use  of  potentially 
unsafe  modes  or  alert  the  operator  that  a  particular 
mode  may  be  hazardous. 

The  current  INCOMMANDS  CDSC  does 
not  prevent  the  use  of  potentially  unsafe 
modes  or  alert  the  operator  that  a  particular 
mode  may  be  hazardous. 

Major 

7,4  PROVIDE  SYSTEM  RESPONSl 

E  AND  FEEDBACK 

Not  Compliant 

7.4.1  -7.4.7 

These  items  deal  with  feedback  that  the  system 
provides  to  the  operator. 

The  current  INCOMMANDS  CDSC  does 
not  provide  feedback  to  the  operator  after 
the  operator  provides  an  input  to  the 
system. 

Major 

7,5  SUPPORT  IDENTIFICATION  AND  MANAGEMENT  OF  ESS  FAULTS  AND  FAl 

[LURES 

Not  Compliant 

7.5.1  -7.5.6 

These  items  deal  with  the  system  providing  the 
operator  with  visual  identification  of  faults  and 
failures. 

The  current  INCOMMANDS  CDSC  does 
not  show  ESS  faults  and  failures. 

Major 
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Compliance 

Paragraph 

INCOMMANDS  HE  Design  and  Evaluation 
Guide 

INCOMMANDS  CDSC 

Severity 

Rating 

8  CLASS-SPECIFIC  ESS  GUIDELINES 

8.2  DECISION  MAKING  AIDS 

8,2,1  ENSURE  APPROPRIATE  IMJ 

PLEMENTATION 

Experimental  -  Out  of 
Scope 

8.2.1. 1 

Decision  making  aids  shall  be  used:  for  managing 
system  complexity;  for  assisting  operators  in 
coping  with  information  overload;  for  focusing  the 
operator’s  attention;  for  assisting  the  operator  in 
accomplishing  time-consuming  activities  more 
quickly;  when  limited  data  results  in  uncertainty; 
for  overcoming  human  limitations  that  are 
associated  with  uncertainty,  the  emotional 
components  of  decision-making,  finite -memory 
capacity,  and  systematic  and  cognitive  biases;  and, 
for  assisting  the  operator  in  allocating  resources, 
managing  detailed  information,  performing 
computations,  and  selecting  and  deciding  among 
alternatives. 

The  performance  measures,  NASA-TLX 
workload  measure,  situation  awareness 
questionnaire  would  need  to  be  used,  and 
the  usability  and  utility  questionnaires 
were  used  to  assess  if  the  INCOMMANDS 
CDSC  assists  the  operator  with  these 
requirements. 

TBD 

8,2,2  SUPPORT  DEC] 

ISION  MAKING  STRATEGIES 

Not  Compliant 

8.2.2.1 

The  decision  making  aid  shall  support  decision 
alternatives. 

The  current  INCOMMANDS  CDSC  does 
not  allow  operators  to  select  their  choice  of 
atomic  action  plans.  The  operator  must  use 
the  atomic  action  plan  generated  by  the 
system. 

Major 

Not  Compliant 

8.2.2.2 

When  more  than  one  alternative  is  available,  the 
decision  making  aid  shall  provide  alternatives  in  a 
recommended  prioritization  scheme  based  on 
mission  and  task  analysis. 

The  current  INCOMMANDS  CDSC  does 
not  provide  alternative  atomic  action  plans. 

Major 

Not  Compliant 

8.2.2.4 

When  the  information  used  by  a  decision  making 

The  current  INCOMMANDS  CDSC  does 

Major 
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Compliance 

Paragraph 

INCOMMANDS  HE  Design  and  Evaluation 
Guide 

INCOMMANDS  CDSC 

Severity 

Rating 

aid  is  derived  or  processed,  the  data  from  which  it 
is  derived  shall  be  either  visible  or  accessible  for 
verification. 

not  show  the  atomic  action  plan  data  from 
which  the  information  used  by  a  decision 
making  aid  is  derived. 

Experimental  -  Out  of 
Scope 

8.2.2.7 

The  support  provided  by  the  decision  making  aid 
shall  be  consistent  with  operator  cognitive 
strategies  and  expectations  (mental  models).  A 
mental  model  is  an  individual’s  understanding  of 
the  processes  underlying  system  operation. 

The  performance  measures  would  need  to 
be  used  to  assess  if  the  INCOMMANDS 
CDSC  is  consistent  with  operator 
cognitive  strategies  and  expectations. 

TBD 

Experimental  -  Out  of 
Scope 

8.2.2.9 

The  decision  making  aid  shall  minimize  queries  by 
the  operators  for  information. 

The  performance  measures  and  the 
usability  and  utility  questionnaires  will 
assess  if  the  INCOMMANDS  CDSC 
minimizes  queries  from  operators  for 
information. 

Minor 

8.2.3  KEEP  OPERATORS  IN  CONI 

fROL 

Not  Compliant 

8.2.3.2 

The  decision  making  aid  shall  not  be  able  to  veto 
operator  actions  leaving  the  operator  without  means 
to  override  or  violate  rules  that  govern  the  decision 
making  aid  unless  there  is  not  enough  time  for  the 
operator  to  make  a  decision. 

The  current  INCOMMANDS  CDSC  does 
not  allow  operators  to  veto  or  override  the 
system-generated  atomic  action  plans. 

Major 

Not  Compliant 

8.2.3.3 

The  operator  shall  be  able  to  initiate  (i.e.,  over-ride) 
the  automation  of  tasks  even  when  a  task  has  been 
designated  to  be  decision  making  aid-initiated. 

The  current  INCOMMANDS  CDSC  does 
not  allow  operators  to  initiate  the 
automation  of  tasks. 

Major 

Not  Compliant 

8.2.3.4 

The  decision  making  aid  shall  assist,  rather  than 
replace,  human  decision  makers  by  providing  data 
for  making  judgments  rather  than  commands  that 
the  operator  must  execute. 

The  current  INCOMMANDS  CDSC  does 
not  allow  operators  to  override  the  atomic 
action  plans  generated  by  the  decision 
support  system. 

Major 

Not  Compliant 

8.2.3.6 

The  decision  making  aid  shall  accept  direction 
from  the  operators  on  which  problem  solving 
strategy  to  employ  when  alternative  strategies  are 

The  current  INCOMMANDS  CDSC  does 
not  allow  operators  to  direct  the  decision 
making  aid  as  to  which  problem-solving 

Major 
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Compliance 

Paragraph 

INCOMMANDS  HE  Design  and  Evaluation 
Guide 

INCOMMANDS  CDSC 

Severity 

Rating 

available. 

strategy  to  employ. 

Not  Compliant 

8.2.3.7 

Automated  tasks  or  functions  shall  not  be  able  to 
jeopardize  safety  or  make  a  difficult  situation 
worse. 

The  current  INCOMMANDS  CDSC  does 
not  allow  operators  to  override  the  atomic 
action  plans  and  this  could  possibly  make 
a  difficult  situation  worse. 

Major 

Not  Compliant 

8.2.3.8 

When  an  operator  might  need  to  operate  in  out-of¬ 
tolerance  conditions,  then  a  deliberate  overriding 
action  shall  be  possible. 

The  current  INCOMMANDS  CDSC  does 
not  have  a  deliberate  overriding  action  that 
the  operator  can  operate  in  out-of¬ 
tolerance  conditions. 

Major 

8,2.4  MAXIMIZE  OPERATOR  SITUATION  AWARENESS  BY  INCREASING  SYSTEM  TRANSPARENCY 

Not  Compliant 

8.2.4.1 

Processed  data  shall  be  accessible. 

The  current  INCOMMANDS  CDSC  does 
not  allow  the  operator  to  access  or  see  all 
processed  data. 

Major 

Not  Compliant 

8.2.4.4 

The  decision  aid  shall  give  the  operator  access  to 
procedural  information  used  by  the  aid. 

The  current  INCOMMANDS  CDSC  does 
not  give  the  operator  access  to  procedural 
information  used  by  the  aid. 

Major 

Not  Compliant 

8.2.4.5 

When  the  decision  making  aid  provides 
explanations  to  the  operator,  it  shall  supply  a  short 
explanation  initially,  with  the  ability  to  make 
available  more  detail  at  the  operator’s  request, 
including  access  to  process  information  or  an 
explanation  for  the  rules,  knowledge-basis,  and 
solutions  used  by  the  decision  aid. 

The  current  INCOMMANDS  CDSC  does 
not  provide  a  short  explanation  and  does 
not  make  available  more  detail  at  the 
operator’s  request. 

Major 

8.3  CONTROL  AND  ACTION  AIDS 

8.3,1  KEEP  OPERATORS  ‘IN-THE-LOOP’ 

Not  Compliant 

8.3. 1.2 

The  Control  and  Action  Aid  shall  provide  the 
operator  with  an  appropriate  range  of  control 
options  that  are  flexible  enough  to  accommodate 

The  current  INCOMMANDS  CDSC  does 
not  provide  the  operator  with  an 
appropriate  range  of  control  options  that 

Major 
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INCOMMANDS  HF  Design  and  Evaluation 
Guide 

INCOMMANDS  CDSC 

Severity 

Rating 

the  full  range  of  operating  conditions  for  which  it 
was  certified. 

are  flexible  enough  to  accommodate  the 
full  range  of  operating  conditions 

Not  Compliant 

8.3. 1.3 

To  promote  sufficient  levels  of  operator  situation 
awareness  of  the  Control  and  Action  Aid,  the 
operator  shall  be  given  immediate  feedback  to 
command  and  control  orders. 

The  current  INCOMMANDS  CDSC  does 
provide  the  operator  with  immediate 
feedback  to  command  and  control  orders. 

Major 

Not  Compliant 

8.3. 1.4 

Override  and  backup  control  alternatives  shall  be 
available  for  automated  tasks  that  are  critical  to  the 
integrity  of  the  system  or  when  lives  depend  on  the 
system. 

The  current  INCOMMANDS  CDSC  does 
not  allow  the  operator  to  override  and 
control  alternatives  for  automated  tasks 
that  are  critical  to  the  integrity  of  the 
system  or  when  lives  depend  on  the 
system. 

Major 

Not  Compliant 

8.3. 1.5 

The  operator  shall  be  able  to  initiate  and  control  the 
direction  and  pace  of  the  tasks  and/or  functions  of 
the  Control  and  Action  Aid  until  the  point  at  which 
operator  goals  have  been  met 

The  current  INCOMMANDS  CDSC  does 
allow  the  operator  to  initiate  and  control 
the  direction  and  pace  of  the  tasks  and/or 
functions  of  the  Control  and  Action  Aid 
until  the  point  at  which  operator  goals 
have  been  met. 

Major 

Not  Compliant 

8.3. 1.6 

Information  for  backup  or  override  capability  shall 
be  readily  accessible. 

The  current  INCOMMANDS  CDSC  does 
allow  the  operator  to  access  information  to 
either  backup  or  override  the  capability  of 
the  system. 

Major 

Not  Compliant 

8.3. 1.7 

The  Control  and  Action  Aid  shall  be  designed  so 
that  operators  are  involved  in  active  control  and 
monitoring  rather  than  just  passive  monitors. 

The  current  INCOMMANDS  CDSC  does 
not  allow  the  operator  to  interact  very 
much  with  the  system. 

Major 

Not  Compliant 

8.3. 1.8 

Allow  reversal  of  operator  actions  (e.g.  ‘undo’  or 
‘cancel’  function)  and  give  clear  indications  how 

The  current  INCOMMANDS  CDSC  does 
allow  operators  to  undo  or  cancel 
functions. 

Major 
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reversal  can  be  achieved. 

10  WINDOWS 

10.2  SYSTEM  WINDOWS 

10.2,1  STATUS  BAR 

Not  Compliant 

10.2.1.4 

A  digital  clock  shall  be  displayed  to  the  right  end  of 
the  status  bar,  showing  the  Date/Time  Group. 

There  is  no  digital  clock  displayed  on  the 
right  end  of  the  status  bar. 

Minor 

Not  Compliant 

10.2.1.5 

An  alert  (and  messages)  indicator  (to  notify  the 
operator  that  alerts  are  present)  is  displayed  on  the 
left  end  of  the  status  bar. 

There  is  no  alert  indicator  on  the  left  end 
of  the  status  bar. 

Minor 

Not  Compliant 

10.2.1.6 

The  notification  of  an  alert  shall  indicate  the 
priority  of  the  alert,  if  available. 

There  is  no  alert  indicator  on  the  left  end 
of  the  status  bar. 

Minor 

Not  Compliant 

10.2.1.7 

The  contents  of  an  alert  (and  message)  shall  be 
displayed  in  an  area  dedicated  for  that  purpose  near 
and  below  the  alert  notification  indicator.) 

There  is  no  alert  indicator  on  the  left  end 
of  the  status  bar. 

Minor 

Not  Compliant 

10.2.1.8 

System  alerts  shall  be  identified  distinctly  from 
operational  alerts  in  the  alert  notification  area. 

There  is  no  alert  indicator  on  the  left  end 
of  the  status  bar. 

Minor 

10,2,2  OPTIONAL  DI 

SPLAY  OF  CLASSIFICATION 

Out  of  Scope 

10.2.2.1  - 
10.2.2.11 

This  section  covers  how  classified  windows  shall 
be  displayed. 

The  current  INCOMMANDS  CDSC  does 
not  display  classified  windows. 

Minor 

10.3  PRIMARY  WINDOWS 

10.3.4  FIXED  WINDOWS 

10,3.4,1  FIXED  WINDOWS  CHARACTERISTICS 

Not  Compliant 

10.3.4.1.2 

Fixed  windows  shall  be  automatically  loaded  upon 
system  initialization  and  cannot  be  closed  or 
minimized. 

The  fixed  windows  currently  have  a 
minimize/maximize/restore  control. 

Minor 
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Not  Compliant 

10.3.4.1.3 

Fixed  windows  cannot  be  re-sized  or  re-located  by 
the  user. 

The  fixed  windows  currently  have  a  re-size 
control. 

Minor 

Not  Compliant 

10.3.4.1.6 

The  title  bar  and  sizing  frame  are  removed  from 
fixed  windows  since  they  cannot  be  closed,  moved 
or  sized. 

The  title  bar  and  sizing  frame  have  not 
been  removed  from  the  fixed  window. 

Minor 

11  WINDOW  DESIGN  GUIDANCE 

11,2  ARRANGING 

[FORMATION  TO  MATCH  USER  ACTIONS 

Not  Compliant 

11.2.1 

A  window  shall  be  designed  so  users  can 
manipulate  objects  in  ways  that  support  task 
performance. 

The  current  INCOMMANDS  CDSC  does 
not  allow  operators  to  manipulate  objects 
in  ways  that  support  task  performance. 

Major 

12  WINDOW  NAVIGATION  AND  SELECTION 

12.1  WINDOW  NAVIGATION 

12.1.3  ASSIGNING  FOCUS  TO  A  WINDOW  WITH  A  POINTING  DEVICE  AND  THl 

E  KEYBOARD 

Not  Compliant 

12.1.3.1 

Users  assign  focus  by  moving  the  pointer  into  a 
window  and  clicking  the  S  button. 

The  current  INCOMMANDS  CDSC  does 
not  allow  operators  to  click  the  S  button  to 
assign  focus. 

Minor 

Not  Compliant 

12.1.3.2 

If  users  click  in  an  empty  window  area,  the  window 
frame  shall  highlight  and  the  window  shall  be 
raised  to  the  front. 

When  clicked  on,  the  windows  frame  on 
the  current  INCOMMANDS  CDSC  does 
not  highlight  and  the  window  does  not 
raise  to  the  front. 

Minor 

Not  Compliant 

12.1.3.3 

If  users  click  in  the  title  bar,  the  frame  shall 
highlight  and  the  window  shall  be  raised  to  the 
front  and  shall  receive  input  focus. 

When  the  title  bar  is  clicked  on  the  current 
INCOMMANDS  CDSC,  the  frame  is  not 
highlighted  and  raised  to  the  front. 

Minor 

Not  Compliant 

12.1.3.4 

If  users  click  on  an  object  within  a  window,  the 
window  frame  shall  highlight,  the  window  shall  be 
raised,  and  an  object  shall  be  selected. 

When  an  object  is  clicked  on  within  a 
window  on  the  current  INCOMMANDS 
CDSC,  the  window  frame  does  not 
highlight  and  raise. 

Minor 

DRDC  Toronto  CR  2009-041 


53 


Compliance 

Paragraph 

INCOMMANDS  HE  Design  and  Evaluation 
Guide 

INCOMMANDS  CDSC 

Severity 

Rating 

Not  Compliant 

12.1.3.5 

A  window  is  raised  by  clicking  anywhere  in  the 
window.  The  window  is  then  moved  to  the  top  of 
the  window  hierarchy  (except  for  any  toolbars  or 
pallets  associated  with  the  window  being  raised) 
and  is  given  input  focus.  A  window  may  also  be 
raised  by  selecting  it  from  the  open  window  menu. 

The  windows  on  the  current 
INCOMMANDS  CDSC  do  not  raise  when 
clicked  on. 

Minor 

Not  Compliant 

12.1.3.6 

<Alt>  +  <Tab>  and  <Alt>  +  <Shift>  +  <Tab>  shall 
move  the  focus  forward  and  backward  through 
window  families  (i.e.,  primary  windows  and 
window  icons). 

The  Current  INCOMMANDS  CDSC  does 
not  allow  operators  to  input  these 
commands. 

Minor 

Not  Compliant 

12.1.3.7 

<Alt>  +  <F6>  and  <Alt>  +  <Shift>  +  <F6>  move 
the  focus  forward  and  backward  through  windows 
in  a  family. 

The  Current  INCOMMANDS  CDSC  does 
not  allow  operators  to  input  these 
commands. 

Minor 

12.2  NAVIGATION  WITHIN  WINDOWS 

12.2.1  POINTING  DEVICE  NAVIGATION  FOR  CONTROLS 

Not  Compliant 

12.2.1.1 

Placing  the  pointer  on  a  control  and  clicking  the  S 
button  shall  move  the  location  cursor  to  the  object 
and  shall  give  the  object  focus. 

The  current  INCOMMANDS  CDSC  does 
not  allow  operators  to  input  this  command. 

Minor 

Not  Compliant 

12.2.1.2 

Pressing  the  <Ctrl>  key  and  clicking  the  S  button 
on  an  object  shall  select  the  object  and  keep  the 
current  object  selected. 

The  current  INCOMMANDS  CDSC  does 
not  allow  operators  to  input  this  command. 

Minor 

Not  Compliant 

12.2.1.3 

Autoscrolling  shall  be  available  when  the  pointer  is 
on  a  scrollable  control  such  as  a  text  block  or  a  list. 

The  current  INCOMMANDS  CDSC  does 
not  allow  operators  to  input  this  command. 

Minor 

Not  Compliant 

12.2.1.4 

The  means  shall  be  provided  to  readily  move  the 
cursor  to  the  head  or  the  foot  (end)  of  a  file. 

The  current  INCOMMANDS  CDSC  does 
not  allow  operators  to  input  this  command. 

Minor 

12.2.2  KEYBOARD  N 

AVIGATION  FOR  CONTROLS 

Not  Compliant 

12.2.1  - 
12.2.2.15 

The  current  INCOMMANDS  CDSC  does 
not  allow  operators  to  input  any  of  the 
Keyboard  Navigations 

Minor 
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12.2.3  KEYBOARD  N 

AVIGATION  FOR  GRAPHIC  OBJECTS 

Not  Compliant 

12.2.3.1 

Navigation  among  graphics  objects  shall  use  the 
same  key  bindings  as  navigation  among  controls. 

The  current  INCOMMANDS  CDSC  does 
not  allow  operators  to  input  these 
commands. 

Minor 

12.3  OBJECT  SELECTION 

12.3.1  POINTING  DEVICE  SELECTION  METHODS 

Not  Compliant 

12.3.1.2 

To  select  multiple  objects,  click  the  S  button  on 
objects  one  at  a  time  while  holding  down  the 
<Ctrl>  key.  The  objects  are  highlighted  and 
selected.  (TBM,  MSWUE). 

The  current  INCOMMANDS  CDSC  does 
not  allow  operators  to  input  this  command. 

Minor 

Not  Compliant 

12.3.1.3 

Where  text  has  been  specified  to  become  the 
subject  of  control  entries  (e.g.,  for  underlining, 
bolding,  moving,  copying,  or  deleting),  the  affected 
segment  of  text  shall  be  highlighted  to  indicate  its 
boundaries. 

The  current  INCOMMANDS  CDSC  does 
not  allow  operators  to  input  control  entries. 

Minor 

Not  Compliant 

12.3.1.4 

To  select  a  range  of  contiguous  objects,  the  pointer 
shall  be  positioned  on  the  first  object  in  the  range  to 
be  selected,  and  then  the  S  button  shall  be  selected 
to  set  the  anchor  for  the  range.  Any  other  object 
that  was  selected  shall  be  deselected.  The  pointer 
shall  be  dragged  until  it  is  on  the  last  object  in  the 
range,  and  the  S  button  shall  be  released  to 
complete  the  selection. 

The  current  INCOMMANDS  CDSC  does 
not  allow  operators  to  select  a  range  of 
contiguous  objects. 

Minor 

Not  Compliant 

12.3.1.5 

To  extend  a  range  selection  the  user  shall  position 
the  pointer  on  the  object  that  shall  be  the  last  one  in 
the  selection,  and  hold  down  the  <Shift>  key  while 
clicking  the  S  button  on  the  pointing  device.  The 
objects  in  the  revised  selection  range  (defined  from 
the  original  anchor  to  the  current  pointer  position  in 

The  current  INCOMMANDS  CDSC  does 
not  allow  operators  to  carry  out  this 
operation. 

Minor 
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one-dimensional  collections,  and  defined  by  the 
diagonal  from  the  anchor  to  the  current  pointer 
position  in  two  dimensional  collections)  shall  be 
reselected,  and  any  elements  removed  from  the 
selection  shall  return  to  normal  appearance. 

Not  Compliant 

12.3.1.6 

To  add  or  remove  a  non-contiguous  object  to  or 
from  a  range  selection,  the  pointer  shall  be 
positioned  on  the  object,  and  then  the  <Ctrl>  key 
shall  be  held  down  while  clicking  the  S  button  on 
the  pointing  device.  If  previously  unselected,  the 
object  shall  be  selected  and  highlighted.  If 
previously  selected,  the  object  is  deselected  and 
shall  return  to  a  normal  appearance.  The  other 
elements  in  the  selection  shall  remain  highlighted. 

The  current  INCOMMANDS  CDSC  does 
not  allow  operators  to  select  a  range  of 
contiguous  objects. 

Minor 

Not  Compliant 

12.3.1.7 

A  bounding  box  shall  appear  when  dragging  the 
pointer  over  elements  in  a  two  dimensional 
collection. 

A  bounding  box  does  not  appear  when 
dragging  the  pointer  over  elements  in  a 
two  dimensional  collection. 

Minor 

12.3.2  KEYBOARD  SELECTION  M 

lETHODS 

Not  Compliant 

12.3.2.1  - 
12.3.2.10 

The  current  INCOMMANDS  CDSC  does 
not  allow  operators  to  input  any  of  the 
Keyboard  Selection  Methods 

Minor 

12.3.3  OTHER  TYPES  OF  SELECTION 

Not  Compliant 

12.3.3.1  - 
12.3.3.6 

The  current  INCOMMANDS  CDSC  does 
not  allow  operators  to  select  any  other 
types  of  inputs  from  the  keyboard 

Minor 

12.5  INTERACTIVE  CONTROL 

12.5.1  OBJECT-ACT] 

[ON  SELECTION 

Not  Compliant 

12.5.1.1 

Users  shall  first  select  an  object,  and  then  they  shall 
select  an  action  to  perform  on  that  object. 

The  current  INCOMMANDS  CDSC  does 
not  allow  operators  to  perform  actions  on 

Major 
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selected  objects. 

12,5.2  USER  CONTROL  OF  INTERACTION 

Not  Compliant 

12.5.2.1 

Applications  shall  execute  an  action  only  in 
response  to  explicit  user  input. 

The  current  INCOMMANDS  CDSC  does 
not  allow  operators  to  input  commands. 

Major 

Not  Compliant 

12.5.2.2 

Users  may  take  actions  that  will  interrupt  or 
terminate  a  process. 

The  current  INCOMMANDS  CDSC  does 
not  allow  operators  to  terminate  processes. 

Major 

12.5.3  IMMEDIATE  FEEDBACK 

Not  Compliant 

12.5.3.1 

Control  feedback  responses  to  correct  user  input 
shall  consist  of  changes  in  state  or  value  of  those 
display  elements  which  are  being  controlled  and 
shall  be  presented  in  an  expected  and  logically 
natural  form. 

The  current  INCOMMANDS  CDSC  does 
not  provide  control  feedback  responses. 

Major 

Not  Compliant 

12.5.3.2 

When  users  take  an  action,  there  shall  be  an 
immediate  and  visible  response  to  the  action. 

The  current  INCOMMANDS  CDSC  does 
not  accept  operator  inputs  so  therefore, 
there  are  no  immediate  and  visible 
responses  to  the  action. 

Major 

Not  Compliant 

12.5.3.3 

A  visible  response  shall  occur  even  if  the  result 
cannot  be  displayed  immediately. 

The  current  INCOMMANDS  CDSC  does 
not  accept  operator  inputs  so  therefore,  no 
visible  responses  is  available. 

Major 

Not  Compliant 

12.5.3.4 

An  application  shall  provide  visual  cues  that 
indicate  when  it  can  accept  input,  when  it  is 
temporarily  unavailable,  and  when  it  is  unavailable 
during  extended  processing. 

The  current  INCOMMANDS  CDSC  does 
not  accept  operator  inputs  so  no  visual 
cues  that  indicate  when  the  system  can 
accept  input,  when  it  is  temporarily 
unavailable,  and  when  it  is  unavailable 
during  extended  processing  are  available. 

Major 

Not  Compliant 

12.5.3.6 

If  an  operation  requires  several  actions,  users  shall 
be  prompted  with  the  actions  to  take. 

The  current  INCOMMANDS  CDSC  does 
not  provide  operators  with  prompts  for 
actions  that  are  to  be  taken. 

Major 

Not  Compliant 

12.5.3.7 

Applications  shall  ignore  user  actions  made  during 

The  current  INCOMMANDS  CDSC  does 

Major 
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periods  when  input  cannot  be  accepted. 

not  accept  operator  inputs  so  the  system 
cannot  ignore  actions  made  during  periods 
when  input  cannot  be  accepted. 

Not  Compliant 

12.5.3.8 

The  pointing  device  and/or  the  keyboard  shall  be 
disabled  when  input  may  be  destructive. 

The  current  INCOMMANDS  CDSC  does 
not  disable  the  keyboard  and/or  pointing 
device  when  input  may  be  destructive. 

Major 

Not  Compliant 

12.5.3.9 

Although  an  application  shall  not  allow  users  to 
override  disabling,  users  shall  be  able  to  stop  a 
process  if  desired  (e.g.,  by  selecting  a  cancel,  or 
equivalent,  push  buhon). 

The  current  INCOMMANDS  CDSC  does 
not  allow  operators  to  stop  processes. 

Major 

Not  Compliant 

12.5.3.10 

The  current  value  of  any  parameter  or  variable  with 
which  the  user  is  interacting  shall  be  displayed. 

The  current  INCOMMANDS  CDSC  does 
not  display  the  current  value  of  any 
parameter  or  variable. 

Major 

12.5.4  SYSTEM  RESPONSE  TIME 

Not  Compliant 

12.5.4.1  - 
12.5.4.22 

The  current  INCOMMANDS  CDSC  has 
limited  capability  to  allow  operator  to 
interact  with  the  system 

Major 

12.5.5  ERROR  DETECTION 

Not  Compliant 

12.5.5.1  - 
12.5.4.11 

The  current  INCOMMANDS  CDSC  does 
not  provide  any  error  messages  to 
operators. 

Major 

12.5.6  UNDO  CAPABILITY 

Not  Compliant 

12.5.6.1  - 
12.5.6.7 

The  current  INCOMMANDS  CDSC  does 
not  allow  the  operator  to  undo  any  actions. 

Major 

13  CONTROLS 

13.1  CONTROL  CHARACTERISTICS 

Not  Compliant 

13.1.6 

Controls  that  are  temporarily  unavailable  shall  be 
dimmed  and  shall  not  be  available  for  selection. 

The  current  INCOMMANDS  CDSC  does 
dim  temporarily  unavailable  controls  and 

Major 
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make  them  unavailable  for  selection. 

Not  Compliant 

13.1.7 

Controls  that  are  never  available  to  users  shall  not 
appear  in  a  window. 

The  current  INCOMMANDS  CDSC 
shows  all  of  the  controls  to  the  operator 
even  though  some  are  not  yet  functioning. 

Major 

13.5  PUSH  BUTTONS 

13,5,5  ACTIVATING  A  PUSH  BUTTON 

Not  Compliant 

13.5.5.2 

The  <Space>  key  (and  <Select>  if  available)  shall 
select  a  push  button  from  the  keyboard  when  the 
location  cursor  is  on  a  push  button. 

The  current  INCOMMANDS  CDSC  does 
not  allow  the  operator  to  input  this  action. 

Minor 

Not  Compliant 

13.5.5.3 

When  a  push  button  is  selected,  it  shall  highlight 
and  the  action  it  represents  shall  be  executed. 

When  a  push  button  is  selected,  it  does  not 
highlight. 

Minor 

13.6  TOOLBARS 

13.6.3  TOOLBAR  WI 

NDOWS 

Not  Compliant 

13.6.3.1 

All  toolbars  shall  be  able  to  float  by  selecting  any 
region  inside  the  toolbar  and  moving  it  to  another 
display  location 

The  toolbars  do  not  float  and  cannot  be 
moved  to  another  display  location. 

Minor 

Not  Compliant 

13.6.3.2 

The  floating  toolbar  window  shall  then  be  able  to 
be  closed,  sized,  or  moved  to  any  other  display 
location  like  a  normal  window 

The  toolbar  does  not  float. 

Minor 

Not  Compliant 

13.6.3.5 

An  application  action  icon  palette  or  toolbar  shall 
be  able  to  be  repositioned  or  removed  from  the 
window  if  desired  by  the  user 

The  tools  cannot  be  repositioned  or 
moved. 

Minor 

17  DATA  DISPLAY  AND  ENTRY 

17.3  DATA  DISPLAY 

17.3,7  NUMBERS 

Not  Compliant 

17.3.7.1 

When  displaying  numbers  the  number  zero  shall 
have  a  slash  through  it  so  it  is  not  confused  with  a 
capital  "0".  The  letter  L  and  the  digit  1  shall  also 

The  number  zero  on  the  INCOMMANDS 
CDSC  does  not  have  a  slash  through  it. 

Minor 
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be  displayed  so  as  not  to  be  confused  with  each 
other 

17.3.8  UNITS  OF  MEASUREMENT 

Not  Compliant 

17.3.8.5 

The  system  shall  allow  users  to  input  data  in  a 
familiar  unit  of  measure. 

Operators  cannot  input  any  information 
into  the  current  INCOMMANDS  CDSC. 

Minor 

17.3.13  DATA  QUERY 

Not  Compliant 

17.3.13.2 

The  language  shall  allow  the  users  to  specify  the 
data  to  retrieve,  then  display  and  manipulate  it. 

Operators  cannot  retrieve,  display  or 
manipulate  data. 

Major 

Not  Compliant 

17.3.13.3 

Users  shall  be  provided  the  capability  to  request 
data  without  having  to  tell  the  system  how  to  find 
it. 

Operators  cannot  retrieve  data  from  the 
system. 

Major 

Not  Compliant 

17.3.13.4 

Queries  shall  use  operationally  meaningful 
terminology  and  do  not  reflect  how  the  data  are 
stored. 

Operators  cannot  query  the  system  for 
data. 

Major 

Not  Compliant 

17.3.13.5 

Users  may  construct  simple  and  complex  queries, 
create  predefined  queries  and  save,  retrieve,  and 
execute  these  queries. 

Operators  cannot  query  the  system  for 
data. 

Major 

Not  Compliant 

17.3.13.6 

The  language  shall  permit  alternate  forms  of  the 
same  query  using  natural  language. 

Operators  cannot  query  the  system  for 
data. 

Major 

Not  Compliant 

17.3.13.7 

Users  shall  be  prompted  to  confirm  a  query  if  data 
retrieval  time  will  be  excessive. 

Operators  cannot  query  the  system  for 
data. 

Major 

18  SPECIAL  FUNCTIONS  AND  FORMATS 

18.2  DATE/TIME  AND  LATITUDE/LONGITUDE 

Not  Compliant 

18.2.1 

All  dates  shall  be  presented  to  and  supplied  by  the 
operator  in  the  DD  MMM  YY  (10  MAY  68) 
format.  The  characters  of  the  month  shall  be 
capitalized.  The  spaces  between  day  and  month  and 
month  and  year  may  be  omitted.  If  the  day  or  year 

Dates  are  not  presented  in  the  DD  MMM 
YY  format. 

Minor 
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is  a  single  digit  it  must  be  preceded  by  a  zero.  The 
notation  OIJANOO  would  mean  January  1,  2000. 

Not  Compliant 

18.2.2 

Time  shall  be  presented  to  and  supplied  by  the 
operator  in  the  HH:MM:SSZ  (09:38:30Z)  format. 

HH  is  the  hour  in  a  24-hour  day,  MM  is  the  minute 
with  a  leading  zero  if  necessary,  and  SS  is  the 
second  with  the  leading  zero  if  necessary.  Z  is  the 
time  zone  with  Zulu  time  as  the  default.  The 
seconds  are  optional.  All  colons  are  required  and 
the  Zulu  (Z)  shall  be  capitalized. 

Times  is  not  presented  in  the  HH:MM:SSZ 
format. 

Minor 

Not  Compliant 

18.2.3 

The  display  shall  include  Zulu  time  and  local  time; 
Zulu  time  shall  be  presented  above  local  time. 

The  display  does  not  include  Zulu  time. 

Minor 

Not  Compliant 

18.2.5 

If  the  operator  must  know  the  time  in  multiple  time 
zones,  the  application  shall  provide  a  separate  time 
for  each  zone  or  provide  the  operator  the  ability  to 
change  the  time  zone  displayed 

Time  in  multiple  time  zones  is  not 
displayed. 

Minor 

Not  Compliant 

18.2.6 

A  date  time  group  shall  be  presented  to  and 
supplied  by  the  operator  in  the  DD  HH:MM:SSZ 
MMM  YY  format  (seconds  are  optional)  where  DD 
is  the  day  with  the  leading  zero  if  necessary,  HH  is 
the  hour  with  a  leading  zero  if  necessary,  MM  is 
the  minute  with  a  leading  zero  if  necessary,  SS  is 
the  second  with  a  leading  zero  if  necessary,  and  Z 
is  the  time  zone  with  Zulu  as  the  default  with 

MMM  as  the  abbreviated  month  and  YY  is  the  last 
two  digits  of  the  year.  All  colons  and  spaces  are 
required 

The  date  time  group  is  not  presented  in  the 
DD  HH:MM:SSZ  MMM  YY  format. 

Minor 

22  COLOUR 

22.2  COLOUR  SETS 
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Compliance 

Paragraph 

INCOMMANDS  HF  Design  and  Evaluation 
Guide 

INCOMMANDS  CDSC 

Severity 

Rating 

Not  Compliant 

22.2.5 

Users  shall  be  able  to  select  from  between  colour 
sets:  One  set  shall  be  provided  for  each  of  the 
lighting  conditions  in  the  operations  room  (e.g., 
normal  operations  room  lighting  and  dark 
adaptation  lighting. 

Operators  do  not  have  the  option  of 
selecting  colour  sets. 

Minor 

Not  Compliant 

22.2.6 

The  operator  shall  have  the  options  to  select 
between  colour  sets  for  each  applicable  lighting 
condition  but  shall  not  otherwise  adjust  the 
individual  colours  in  the  display. 

Operators  do  not  have  the  option  of 
selecting  colour  sets. 

Minor 

22.3  SPECIFIC  COLOUR  USE 

Not  Compliant 

22.3.9 

Symbology  shall  be  colour  coded  using 

conventions  including  the  following  unless 

superseded  by  NTDS  or  other  explicitly  specified 
symbology  conventions: 

♦  hostile:  red  (or  magenta) 

♦  friend:  blue  (or  cyan) 

♦  unknown:  yellow 

♦  neutral:  green. 

The  symbology  colour  coding  on  the 

Threat  Evaluation  View  does  not  follow 
NTDS  conventions. 

Major 

Not  Compliant 

22.3.10 

The  Halifax  Class  CCS  specifies  white  symbology 
to  denote  computer-identified  contacts.  The  CCS 
specification  shall  be  followed  unless  superseded 
by  specified  symbology  conventions. 

Computer-identified  contacts  on  the  Threat 
Evaluation  View  are  not  depicted  with 
white  symbology. 

Major 

22.4  COLOUR  GUIDELINES 

Not  Compliant 

22.4.1 

Colour  coding  for  symbology  shall  be  consistent 
with  NTDS  conventions,  with  the  caveat  as 
described  in  Paragraph  and  repeated  in  the  UCA 
note  below.  A  sample  of  NTDS  conventions  is 

The  colour  coding  for  symbology  on  the 
Threat  Evaluation  View  is  not  consistent 
with  NTDS  conventions. 

Major 
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Compliance 

Paragraph 

INCOMMANDS  HF  Design  and  Evaluation 
Guide 

INCOMMANDS  CDSC 

Severity 

Rating 

provided  in  Appendix  111:  NTDS  Symbology. 
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Annex  B  Informed  Voluntary  Consent  Form 


Research  Project  Title: 
Principal  Investigator: 
Co-Investigator: 

DRDC  Co-Investigators: 


Evaluation  of  the  INCOMMANDS  Prototype 
Kevin  Baker  (CAE  Professional  Services,  Canada) 
Lisa  Hagen  (CAE  Professional  Services,  Canada) 
Sharon  McFadden  and  Wenbi  Wang,  DRDC  Toronto 


I,  _ ,  hereby  volunteer  to  participate  as  a  subject  in 

the  study,  “INCOMMANDS”.  1  have  read  the  information  material,  and  have  had  the  opportunity 
to  ask  questions  of  the  study  Investigators.  All  of  my  questions  concerning  this  study  have  been 
fully  answered  to  my  satisfaction.  However,  1  may  obtain  additional  information  about  the 
research  project  and  have  any  questions  about  this  study  answered  by  contacting  Kevin  Baker 
(613-247-0342  x208)  or  Lisa  Hagen  (613-247-0342  x245). 

I  am  aware  that  1  will  participate  in  a  training  session  to  learn  how  to  use  the  system.  Once  1  am 
proficient  and  comfortable  with  the  system,  1  will  complete  a  15-minute  practice  scenario.  After 
completion  of  the  practice  scenario  the  experimental  manager  or  the  SME  on  site  will  ask  me  if  1 
know  how  to  and  1  am  comfortable  with  performing  a  series  of  tasks  that  would  be  required 
during  testing  bearing  in  mind  that  the  experimental  staff  will  not  provide  assistance  to  me 
during  the  experimental  sessions.  Once  1  indicate  that  they  1  am  able  to  perform  these  tasks,  1  will 
be  asked  to  complete  a  consent  form  and  to  provide  some  professional  and  personal  background 
information  (e.g.,  number  of  years  of  experience  as  an  operator).  1  will  then  complete  the  20- 
minute  usability  and  utility  trial  and  then  1  will  fdl  out  a  questionnaire  that  asks  me  to  rate  various 
aspects  of  my  situational  awareness.  Following  completion  of  the  questionnaire,  1  will  be  asked  to 
complete  a  usability  and  a  utility  questionnaire  regarding  the  realism  and  usability  of  the  scenario 
and  the  efficacy  of  the  operator-machine  interface  (e.g.,  the  physical  and  functional  displays). 
Following  completion  of  the  usability  and  the  utility  questionnaires,  1  will  be  debriefed  by  the 
experimental  staff. 

1  have  been  told  that  this  experiment  offers  minimal  risk  to  my  health  and  well-being.  There  is  a 
low  risk  of  eye  fatigue  or  eyestrain,  as  would  be  associated  with  doing  any  visually  intensive  task 
(e.g.  web  searching,  word  processing)  on  computer  display  for  the  period  of  time  used  in  the  test 
sessions.  This  may  manifest  itself  as  eye  discomfort,  dry  or  itchy  eyes,  or  mild  headache. 
However,  the  duration  of  exposure  to  the  visual  displays  will  be  short  and  1  will  have  a  5-minute 
break  between  sessions  on  a  single  day  to  mitigate  this  risk.  1  will  be  encouraged  to  inform 
experimenters  if  I  experience  any  discomfort  or  eyestrain,  or  if  1  have  any  problems  during  the 
investigation. 

1  consider  the  aforementioned  tasks  and  risks  acceptable  and  understand  that  my  participation  in 
this  study  may  involve  risks  that  are  currently  unforeseen  by  the  investigators. 
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[For  Canadian  Forces  (CF)  members  only]  -  1  understand  that  1  am  considered  to  be  on  duty  for 
disciplinary,  administrative  and  Pension  Act  purposes  during  my  participation  in  this  study.  With 
that  said,  this  duty  status  has  no  effect  on  my  right  to  withdraw  from  the  trial  at  any  time  1  wish 
and  it  is  clear  that  no  action  will  be  taken  against  me  for  exercising  this  right.  As  well,  in  the 
unlikely  event  that  my  participation  in  this  study  results  in  a  medical  condition  rendering  me  unfit 
for  service,  1  may  be  released  from  the  CF  and  my  military  benefits  apply. 

1  agree  to  provide  responses  to  questions  that  are  to  the  best  of  my  knowledge  truthful  and 
complete.  1  have  been  advised  that  the  experimental  data  concerning  me  will  be  treated  as 
confidential  and  not  revealed  to  anyone  other  than  the  investigators  without  my  consent  except  as 
data  unidentified  as  to  source.  1  understand  that  my  name  will  not  be  identified  or  attached  in  any 
manner  to  any  publication  arising  from  this  study.  Moreover,  should  it  be  required,  1  allow  the 
experimental  data  to  be  reviewed  by  an  internal  or  external  audit  committee  with  the 
understanding  that  any  summary  information  resulting  from  such  a  review  will  not  identify  me 
personally.  1  am  aware  that  there  will  not  be  any  on-site  medical  coverage  during  the  experiment. 


1  understand  that  1  am  free  to  refuse  to  participate  and  may  withdraw  my  consent  without 
prejudice  or  hard  feelings  at  any  time.  Should  1  withdraw  my  consent,  my  participation  as  a 
participant  will  end  immediately. 


1  have  informed  the  Investigator  that  1  am  currently  a  subject  in  the  following  other  DRDC 

research  project(s): _ 

(cite  Protocol  Number(s)  and  associated  Principal  Investigator(s)),  and  that  1  am  participating  as  a 
subject  in  the  following  reach  project(s)  at  an  institution  other  than  DRDC: 
_  (cite  name(s)  of 


institution(s)). 


1  understand  that  by  signing  this  consent  1  have  not  waived  any  legal  rights  1  may  have  as  a  result 
of  any  harm  to  me  occasioned  by  my  participation  in  this  research  project  beyond  the  risks  1  have 
assumed. 


Volunteer’s  Name: _ Signature: _ 

Date: _ 

Name  of  Witness  to  Signature: _ 

Signature: _ Date: _ 

Section  Flead/Commanding  Officer’s  Signature  (see  Notes  below) 


CO’S  Unit: _ 

Principal  Investigator: _ Signature: 

Date: 
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Notes: 


For  Military  personnel  on  permanent  strength  of  CFEME:  Approval  in  prineiple  by  Commanding 
Offieer  is  given  in  Memorandum  3700-1  (CO  CFEME),  18  Aug  94;  however  members  must  still 
obtain  their  Section  Flead’s  signature  designating  approval  to  participate  in  this  particular 
research  project 

For  other  military  personnel:  All  other  military  personnel  must  obtain  their  Commanding 
Officer’s  signature  designating  approval  to  participate  in  this  research  project. 

FOR  SUBJECT  ENQUIRY  IF  REQUIRED: 

Should  I  have  any  questions  or  concerns  regarding  this  project  before,  during,  or  after 
participation,  I  understand  that  I  am  encouraged  to  contact  CAE  Professional  Services,  300-1135 
Innovation  Drive,  Ottawa,  ON,  K2K  3G7)  or  Defense  R&D  Canada  Toronto  (PO  Box  2000,  1133 
Sheppard  Avenue  West,  Toronto,  Ontario,  Canada,  M3M  3B9).  This  contact  can  be  made  by 
surface  mail  at  these  addresses,  or  in  person,  by  phone,  or  by  email  to  any  of  the  numbers  and 
addresses  listed  below: 

Principal  Investigator: 

Kevin  Baker,  CAE  Professional  Services  (Canada),  613-247-0342  x208,  kevin.baker@cae.com 

Co-Investigator: 

Lisa  Hagen,  CAE  Professional  Services  (Canada),  613-247-0342  x245,  lisa.hagen@cae.com 

DRDC  Co-Investigators: 

Sharon  McFadden,  416-635-2189,  sharon.mcfadden@drdc-rddc.gc.ca 
Wenbi  Wang,  416-635-2000  x3063,  wenbi.wang@drdc-rddc.gc.ca 

Chair,  DRDC  Human  Research  Ethics  Committee  (HREC): 

Dr.  Jack  Landolt,  416-635-2120,  Jack.Landolt@drdc-rddc.gc.ca 

1  understand  that  1  will  be  given  a  copy  of  this  consent  form  so  that  I  may  contact  any  of  the 
above-mentioned  individuals  at  some  time  in  the  future  should  that  be  required. 
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Annex  C  Usability  Questionnaire-TE  Functional  View 


Instructions:  Place  a  mark  in  the  box  that  most  closely  reflects  your  answer. 

1 .  The  size  of  the  threat  list  is  suitable. 

@ 

Strongly 

Disagree 

Disagree 

Border 

Agree 

© 

Strongly 

Agree 

Suggested 

Improvements 

2.  The  size  of  the  swimlanes  is  suitable. 

@ 

Strongly 

Disagree 

Disagree 

Border 

Agree 

© 

Strongly 

Agree 

Suggested 

Improvements 

3.  The  number  of  swimlanes  is  suitable. 

@ 

Strongly 

Disagree 

Disagree 

Border 

Agree 

© 

Strongly 

Agree 

Suggested 

Improvements 

4.  1  found  it  easy  to  toggle  the  x-axis  scale  between  range 

and  time. 

@ 

Strongly 

Disagree 

Disagree 

Border 

Agree 

© 

Strongly 

Agree 

Suggested 

Improvements 

5.  1  found  it  easy  to  expand/reduce  the  x-axis  scale. 

@ 

Strongly 

Disagree 

Disagree 

Border 

Agree 

© 

Strongly 

Agree 

Suggested 

Improvements 
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6.  Titles  and  labels  are  clear. 


@ 

Strongly 

Disagree 

Disagree 

Border 

Agree 

© 

Strongly 

Agree 

Suggested 

Improvements 

7.  The  colours  used  on  the  screen  make  sense  to  me. 

@ 

Strongly 

Disagree 

Disagree 

Border 

Agree 

© 

Strongly 

Agree 

Suggested 

Improvements 

8.  The  size  of  the  lettering  on  the  screen  is  easy  to  read. 


® 

© 

Suggested 

Strongly 

Disagree 

Disagree 

Border 

Agree 

Strongly 

Agree 

Improvements 

9.  1  always  know  where  the  cursor  is  on  the  Threat  Evaluation  Functional  View  display. 


® 

Strongly 

Disagree 

Disagree 

Border 

Agree 

© 

Strongly 

Agree 

Suggested 

Improvements 

10.  The  cursor  travels  at  an  appropriate  speed  on  the  Threat  Evaluation  Functional  View  display. 

® 

Strongly 

Disagree 

Disagree 

Border 

Agree 

© 

Strongly 

Agree 

Suggested 

Improvements 

11.1  can  find  important  items  quickly. 

® 

Strongly 

Disagree 

Disagree 

Border 

Agree 

© 

Strongly 

Agree 

Suggested 

Improvements 
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12.  1  found  the  Threat  Evaluation  Functional  View  easy  to  use. 


@ 

© 

Suggested 

Strongly 

Disagree 

Disagree 

Border 

Agree 

Strongly 

Agree 

Improvements 

13.1  needed  to  learn  a  lot  about  the  Threat  Evaluation  Functional  View  before  1  could  use  it. 


@ 

Strongly 

Disagree 

Disagree 

Border 

Agree 

© 

Strongly 

Agree 

Suggested 

Improvements 

14.  1  felt  comfortable  using  the  Threat  Evaluation  system. 

@ 

Strongly 

Disagree 

Disagree 

Border 

Agree 

© 

Strongly 

Agree 

Suggested 

Improvements 

15.  The  Threat  Evaluation  Functional  View  has  all  the  functions  and  capabilities  1  expect  it  to 
have. 


® 

© 

Suggested 

Strongly 

Disagree 

Disagree 

Border 

Agree 

Strongly 

Agree 

Improvements 

16.  The  organization  of  information  on  the  Threat  Evaluation  Functional  View  screen  was  clear. 


® 

© 

Suggested 

Strongly 

Disagree 

Disagree 

Border 

Agree 

Strongly 

Agree 

Improvements 

17.  The  information  provided  by  the  Threat  Evaluation  Functional  View  was  easy  to  understand. 


® 

© 

Suggested 

Strongly 

Disagree 

Disagree 

Border 

Agree 

Strongly 

Agree 

Improvements 
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18.  1  knew  what  the  Threat  Evaluation  Functional  View  system  was  doing  at  all  times. 


@ 

© 

Suggested 

Strongly 

Disagree 

Disagree 

Border 

Agree 

Strongly 

Agree 

Improvements 

19.  In  general,  the  system  provides  information  users  will  use. 


@ 

Strongly 

Disagree 

Disagree 

Border 

Agree 

© 

Strongly 

Agree 

Suggested 

Improvements 

20.  It  was  simple  to  use  this  system. 

@ 

Strongly 

Disagree 

Disagree 

Border 

Agree 

© 

Strongly 

Agree 

Suggested 

Improvements 

2 1 .  Overall,  1  am  satisfied  with  this  system. 


® 

© 

Suggested 

Strongly 

Disagree 

Disagree 

Border 

Agree 

Strongly 

Agree 

Improvements 

Overall  comments  and  suggestions: 
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Annex  D  Usability  Questionnaire-CPM  Functional  View 


Instructions:  Place  a  mark  in  the  box  that  most  closely  reflects  your  answer. 


1 .  The  size  of  the  local  engagement  plan  is  suitable. 


@ 

© 

Suggested 

Strongly 

Disagree 

Border 

Agree 

Strongly 

Improvements 

Disagree 

Agree 

2.  The  size  of  the  atomic  actions  is  suitable. 


@ 

Strongly 

Disagree 

Disagree 

Border 

Agree 

© 

Strongly 

Agree 

Suggested 

Improvements 

3.  1  found  it  easy  to  expand/reduce  the  x-axis  scale. 

@ 

Strongly 

Disagree 

Disagree 

Border 

Agree 

© 

Strongly 

Agree 

Suggested 

Improvements 

4.  Titles  and  labels  are  clear. 

@ 

Strongly 

Disagree 

Disagree 

Border 

Agree 

© 

Strongly 

Agree 

Suggested 

Improvements 

5.  The  colours  used  on  the  screen  make  sense  to  me. 

@ 

Strongly 

Disagree 

Disagree 

Border 

Agree 

© 

Strongly 

Agree 

Suggested 

Improvements 
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6.  The  size  of  the  lettering  on  the  screen  is  easy  to  read. 


@ 

© 

Suggested 

Strongly 

Disagree 

Disagree 

Border 

Agree 

Strongly 

Agree 

Improvements 

7.  1  always  know  where  the  cursor  is  on  the  Combat  Power  Functional  View  display. 


@ 

© 

Suggested 

Strongly 

Disagree 

Disagree 

Border 

Agree 

Strongly 

Agree 

Improvements 

8.  The  cursor  travels  at  an  appropriate  speed  on  the  Combat  Power  Functional  View  display. 


® 

© 

Suggested 

Strongly 

Disagree 

Disagree 

Border 

Agree 

Strongly 

Agree 

Improvements 

9.  1  can  find  important  items  quickly. 


® 

© 

Suggested 

Strongly 

Disagree 

Disagree 

Border 

Agree 

Strongly 

Agree 

Improvements 

10.  1  found  the  Combat  Power  Functional  View  easy  to  use. 


® 

© 

Suggested 

Strongly 

Disagree 

Disagree 

Border 

Agree 

Strongly 

Agree 

Improvements 

11.1  needed  to  learn  a  lot  about  the  Combat  Power  Functional  View  before  1  could  use  it. 


® 

© 

Suggested 

Strongly 

Disagree 

Disagree 

Border 

Agree 

Strongly 

Agree 

Improvements 
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12.  1  felt  comfortable  using  the  Combat  Power  system. 


@ 

© 

Suggested 

Strongly 

Disagree 

Disagree 

Border 

Agree 

Strongly 

Agree 

Improvements 

13.  The  Combat  Power  Functional  View  has  all  the  functions  and  capabilities  1  expect  it  to  have. 


@ 

© 

Suggested 

Strongly 

Disagree 

Disagree 

Border 

Agree 

Strongly 

Agree 

Improvements 

14.  The  organization  of  information  on  the  Combat  Power  Functional  View  screen  was  clear. 


® 

© 

Suggested 

Strongly 

Disagree 

Disagree 

Border 

Agree 

Strongly 

Agree 

Improvements 

15.  The  information  provided  by  the  Combat  Power  Functional  View  was  easy  to  understand. 


® 

© 

Suggested 

Strongly 

Disagree 

Disagree 

Border 

Agree 

Strongly 

Agree 

Improvements 

16.  1  knew  what  the  Combat  Power  Functional  View  system  was  doing  at  all  times. 


® 

© 

Suggested 

Strongly 

Disagree 

Disagree 

Border 

Agree 

Strongly 

Agree 

Improvements 

17.  In  general,  the  system  provides  information  users  will  use. 


® 

© 

Suggested 

Strongly 

Disagree 

Disagree 

Border 

Agree 

Strongly 

Agree 

Improvements 
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18.  It  was  simple  to  use  this  system. 


@ 

Strongly 

Disagree 

Disagree 

Border 

Agree 

© 

Strongly 

Agree 

Suggested 

Improvements 

19.  Overall,  1  am  satisfied  with  this  system. 

@ 

Strongly 

Disagree 

Disagree 

Border 

Agree 

© 

Strongly 

Agree 

Suggested 

Improvements 

20.  Overall  comments  and  suggestions: 


74 


DRDC  Toronto  CR  2009-041 


Annex  E  Utility  Questionnaire 


Instructions:  Place  a  mark  in  the  box  that  most  closely  reflects  your  answer. 


1.  Overall,  the  use  of  the  Threat  Evaluation  View  improved  my  ability  to  make  decisions  with 
respect  to  the  goals  and  processes  pertaining  to  the  assessment  of  threats  in  the  environment. 


@ 

© 

Suggested 

Strongly 

Disagree 

Border 

Agree 

Strongly 

Improvements 

Disagree 

Agree 

2.  The  Threat  Evaluation  View  provided  me  with  a  more  comprehensive  assessment  of  all 
threats  in  the  environment. 


@ 

© 

Suggested 

Strongly 

Disagree 

Border 

Agree 

Strongly 

Improvements 

Disagree 

Agree 

3.  The  Threat  Evaluation  View  allowed  me  to  form  an  accurate  mental  picture  of  my 
environment. 


® 

© 

Suggested 

Strongly 

Disagree 

Disagree 

Border 

Agree 

Strongly 

Agree 

Improvements 

4.  The  Threat  Evaluation  View  always  made  it  clear  where  the  targets  were  in  relationship  to  the 
ownship. 


® 

© 

Suggested 

Strongly 

Disagree 

Border 

Agree 

Strongly 

Improvements 

Disagree 

Agree 
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5.  It  was  always  clear  when  a  target  made  a  category  jump  (e.g.,  from  a  mid-level  threat  to  a 
high-level  threat). 


@ 

Strongly 

Disagree 

Disagree 

Border 

Agree 

© 

Strongly 

Agree 

Suggested 

Improvements 

6.  The  Threat 

1 

;  Evaluation  Vie 

1  1 

w  helped  me  t( 

1  1 

)  avoid  “tunne 

1  1 

ling”  on  the  hig 

1  1 

best  priority  threat. 

1  1 

@ 

Strongly 

Disagree 


Disagree 


Border 


Agree 


© 

Strongly 

Agree 


Suggested 

Improvements 


7.  It  was  useful  having  the  ability  to  switch  between  a  time  and  a  range  scale  (nautical  miles)  on 
the  Threat  Evaluation  View. 


® 

Strongly 

Disagree 

Disagree 

Border 

Agree 

© 

Strongly 

Agree 

Suggested 

Improvements 

8.  The  information  in  the  roll-over  menu  ^ 
threats  in  the  environment. 

helped  me  to  quickly  gather  information  about  the 

® 

Strongly 

Disagree 

Disagree 

Border 

Agree 

© 

Strongly 

Agree 

Suggested 

Improvements 

9.  The  interface  on  the  Threat  Evaluation  View  allowed  me  to  easily  view  all  of  the  information 
required  to  evaluate  all  the  threats  in  my  environment. 

® 

Strongly 

Disagree 

Disagree 

Border 

Agree 

© 

Strongly 

Agree 

Suggested 

Improvements 
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10.  1  found  that  the  location  of  all  the  information  on  the  Threat  Evaluation  View  interface 
allowed  me  to  quickly  and  easily  evaluate  all  the  threats  in  my  environment. 


@ 

© 

Suggested 

Strongly 

Disagree 

Border 

Agree 

Strongly 

Improvements 

Disagree 

Agree 

11.  If  you  could  make  changes  to  the  interface  on  the  Threat  Evaluation  View,  what  changes 
would  you  make? _ 


12.  The  Threat  Evaluation  View  decreased  my  overall  workload. 


@ 

© 

Suggested 

Strongly 

Disagree 

Disagree 

Border 

Agree 

Strongly 

Agree 

Improvements 

13.  The  Threat  Evaluation  View  would  decrease  my  overall  workload  more  in  a  high- workload 
situation  relative  to  a  low-workload  situation. 


® 

© 

Suggested 

Strongly 

Disagree 

Disagree 

Border 

Agree 

Strongly 

Agree 

Improvements 

14.  The  Threat  Evaluation  View  increased  my  overall  situation  awareness. 


® 

© 

Suggested 

Strongly 

Disagree 

Disagree 

Border 
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Strongly 

Agree 

Improvements 
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15.  The  Threat  Evaluation  View  would  increase  my  overall  situation  awareness  more  in  a  high- 
workload  situation  relative  to  a  low-workload  situation. 
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Strongly 
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Improvements 
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17.  1  would  feel  comfortable  using  the  Threat  Evaluation  View  under  time  pressure. 


® 

© 

Suggested 

Strongly 

Disagree 

Disagree 

Border 

Agree 

Strongly 

Agree 

Improvements 

18.  Overall,  the  Threat  Evaluation  View  is  a  valuable  tool  for  evaluating  threats  in  the 
environment. 


® 

© 

Suggested 

Strongly 

Disagree 

Disagree 

Border 

Agree 

Strongly 

Agree 

Improvements 

19.  Overall,  the  Threat  Evaluation  View  will  make  the  operator’s  job  easier. 


® 

© 

Suggested 

Strongly 

Disagree 

Disagree 

Border 

Agree 

Strongly 

Agree 

Improvements 
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20.  Overall,  the  use  of  the  Combat  Power  Functional  View  would  improve  my  ability  to  make 
decisions  with  respect  to  the  management  of  combat  power. 


@ 
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Suggested 

Strongly 

Disagree 

Disagree 

Border 

Agree 

Strongly 

Agree 

Improvements 

21.  Overall,  the  use  of  the  Combat  Power  Functional  View  would  improve  my  ability  to  make 
decisions  with  respect  to  the  application  of  combat  power. 


@ 

© 

Suggested 

Strongly 

Disagree 

Disagree 

Border 

Agree 

Strongly 

Agree 

Improvements 

22.  The  Combat  Power  Functional  View  would  improve  my  ability  generate  an  ‘optimum’  global 
engagement  plan. 
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Suggested 

Strongly 

Disagree 

Disagree 

Border 

Agree 

Strongly 

Agree 

Improvements 

23.  During  the  execution  of  a  combat  power  plan,  the  Combat  Power  Functional  View  would 
help  me  to  assess  the  outcome  of  an  engagement. 


® 

© 

Suggested 

Strongly 

Disagree 

Disagree 

Border 

Agree 

Strongly 

Agree 

Improvements 

24.  The  interface  on  the  Combat  Power  Functional  View  allowed  me  to  easily  view  all  of  the 
information  required  to  evaluate  the  engagement  plan  present  on  the  display. 


® 

© 

Suggested 

Strongly 

Disagree 

Disagree 

Border 

Agree 

Strongly 

Agree 

Improvements 
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25.  The  interface  on  the  Combat  Power  Functional  View  allowed  me  to  easily  view  all  of  the 
information  required  to  evaluate  whether  the  presented  engagement  plan  would  be  a 
successful  plan. 
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Suggested 

Strongly 

Disagree 

Border 

Agree 

Strongly 

Improvements 

Disagree 

Agree 

26.  1  found  that  the  location  of  all  the  information  on  the  Combat  Power  Functional  View 
interface  allowed  me  to  quickly  and  easily  evaluate  the  engagement  plan  presented  on  the 
display. 
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Suggested 

Strongly 

Disagree 

Disagree 

Border 

Agree 

Strongly 

Agree 

Improvements 

27.  If  you  could  make  changes  to  the  interface  on  the  Combat  Power  Functional  View,  what 
changes  would  you  make? _ 


28.  The  Combat  Power  Functional  View  would  decrease  my  overload  workload. 


® 

© 

Suggested 

Strongly 

Disagree 

Disagree 

Border 

Agree 

Strongly 

Agree 

Improvements 

29.  The  Combat  Power  Functional  View  would  decrease  my  overall  workload  more  in  a  high- 
workload  situation  relative  to  the  low-workload  situation. 


® 
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Suggested 

Strongly 

Disagree 

Disagree 

Border 

Agree 

Strongly 

Agree 

Improvements 
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30.  The  Combat  Power  Functional  View  would  increase  my  overall  situation  awareness. 


@ 

© 

Suggested 

Strongly 

Disagree 

Disagree 

Border 

Agree 

Strongly 

Agree 

Improvements 

31.  The  Combat  Power  Functional  View  would  increase  my  overall  situation  awareness  more  in 
the  high-workload  situation  relative  to  the  low-workload  situation. 


@ 

© 

Suggested 

Strongly 
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Disagree 

Border 
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Strongly 

Agree 

Improvements 

32.  The  presence  of  the  Combat  Power  Functional  View  would  not  interfere  with  the  flow  of 
communication  in  the  operations  room. 
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Strongly 

Disagree 

Disagree 

Border 

Agree 

Strongly 

Agree 

Improvements 

33.  1  would  feel  comfortable  using  the  Combat  Power  Functional  View  under  time  pressure. 


® 

© 

Suggested 

Strongly 

Disagree 

Disagree 

Border 

Agree 

Strongly 

Agree 

Improvements 

34.  Overall,  the  Combat  Power  Functional  View  is  a  valuable  tool  for  managing  and  applying 
combat  power. 
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Disagree 

Disagree 
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Strongly 

Agree 

Improvements 
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35.  Overall,  both  the  Threat  Evaluation  View  and  the  Combat  Power  Functional  View  would 
decrease  my  workload. 
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Disagree 

Border 

Agree 

Strongly 

Agree 

Improvements 

36.  Overall,  both  the  Threat  Evaluation  View  and  the  Combat  Power  Functional  View  would 
increase  my  situation  awareness. 
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Strongly 

Agree 

Improvements 

37.  Should  additional  capabilities  be  added?  Yes _  No 

If  yes,  please  explain _ 


38.  Should  any  of  the  capabilities  be  eliminated  or  changed?  Yes _  No 

If  yes,  please  explain _ 


39.  How  satisfactory  was  the  Threat  Evaluation  View  and  the  Combat  Power  Functional  View 
for  meeting  task  and  time  demands? 


® 

Very 

Unsatisfactory 

Border 

Satisfactory 

© 

Very 

Unsatisfactory 

Satisfactory 
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40.  Overall  comments  and  suggestions: 
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Annex  F  Proposed  Experimental  Plan 


F.1  General 

As  previously  mentioned,  an  experiment  was  initially  proposed  to  investigate  the  causal 
relationships  between  the  INCOMMANDS  CSDC  and  a  serious  of  factors  including  workload, 
situation  awareness,  and  task  performance.  However,  it  was  not  possible  to  carry  out  the 
experiment  within  the  time  frame  of  the  TDP.  The  experimental  plan  created  in  anticipation  of 
this  validation  exercise  has  been  included  in  this  annex  in  the  event  that  it  can  be  performed  at  a 
later  date. 


F.2  Objectives 

The  objective  of  the  investigation  outlined  below  is  to  collect  data  to  be  used  in  the  development, 
validation  and  implementation  of  decision  support  concepts  to  support  the  conduct  of  threat 
evaluation  (TE)  and  combat  power  management  (CPM)  functions  by  Canadian  Forces  (CF) 
personnel  on-board  a  HAFIFAX  Class  frigate. 

The  proposed  experiment  would  be  a  laboratory  based  study  requiring  participants  (experience 
operators)  to  perform  activities  and  make  decisions  to  support  goals  pertaining  to  threat 
evaluation  and  combat  power  management.  Situational  awareness,  workload,  operator  trust,  and 
task  performance  measures  would  be  collected  during  the  experiment. 

F.3  Measures  of  Performance 

For  this  study  several  different  measures  for  assessing  performance  are  proposed.  They  include 
but  are  not  necessarily  limited  to: 

1 .  Situation  awareness 

2.  Workload 

3 .  T ask  relevant  performance 

4.  Trust 

Each  of  these  measures  is  discussed  below  in  the  following  sections. 

F.3.1  Situation  Awareness  (SA) 

It  is  proposed  to  measure  situation  awareness  using  one  or  more  of  the  following  techniques: 

1.  Situation  Awareness  Rating  Technique  (SART):  Participants  complete  the  SART  after  the 
completion  of  each  experimental  trial.  SART  provides  a  validated  and  practical  subjective 
rating  tool  for  the  measurement  of  SA,  based  on  personal  construct  dimensions  associated 
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with  SA.  The  structure  of  the  construct  dimensions  has  been  interpreted  as  comprising  three 
related  conceptual  groups,  which  form  the  principal  dimensions  of  SART,  namely: 

a.  Demand  for  attentional  resources  or  D  (complexity,  variability,  instability); 

b.  Supply  of  attentional  resources  or  S  (arousal,  concentration,  division  of  attention, 
spare  mental  capacity); 

c.  Understanding  of  the  situation  or  U  (information  quality,  information  quantity, 
familiarity). 

The  most  commonly-used  version  of  SART  is  the  14-dimension  version  (see  Figure  18). 
Instead  of  numeric  Likert-scales,  a  graphical  display  of  the  rating  scales  is  utilized,  where  the 
length  of  line  from  the  left  hand  side  of  the  scale  to  the  operator’s  mark  (in  millimetres) 
represents  a  respective  rating  score  for  one  item.  The  possible  range  is  between  0  (“low”)  and 
50  (“high”).  Questions  1,  2,  3  and  4  are  averaged  to  give  a  D  score  (Demand).  Questions  5,  6, 
7,  8  and  9  are  averaged  to  give  an  S  score  (Supply).  Questions  10,  11,  12  and  13  are  averaged 
to  give  a  U  score  (Understanding).  Situation  awareness  in  total  (T)  is  then  calculated  by  U  - 
(D  -  S).  Finally,  Question  14  gives  the  operator’s  confidence  in  their  ratings  of  the  above. 

2.  Change  Blindness:  This  measures  the  operator’s  ability  to  notice  changes  (i.e.,  changed 
entities)  on  their  displays.  It  is  anticipated  that  operators  will  often  fail  to  notice  these 
changes;  however,  in  cases  where  the  changes  are  noticed,  the  time  required  to  notice  them 
should  be  measured.  Change  blindness  is  designed  to  complement  the  SART  measure  by 
providing  an  on-line  metric  of  the  operator’s  SA  for  abrupt  changes  to  their  physical  displays. 
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1.  Demand  on  Attentional  Resources  (capacity) 

How  demanding  were  the  tasks  on  your  attentional  resources?  Were  they  low -  high 

excessively  demanding  (high)  or  minimally  demanding  (low)? 

2.  Instability  of  Session 

How  changeable  was  the  session  (situation)?  Was  it  highly  unstable  and  low -  high 

likely  to  change  suddenly  (high),  or  was  it  very  stable  and  straight  forward 
(low)? 

3.  Complexity  of  Session 

How  complicated  was  the  session?  Was  it  complex  with  many  interrelated  low -  high 

components  (high)  or  was  it  simple  and  straightforward  (low)? 

4.  Variability  of  Session 

How  many  variables  were  changing  in  the  session?  Were  there  a  large  low _  high 

number  of  factors  varying  (high)  or  were  there  very  few  variables  changing 
(low)? 

5.  Supply  of  Attentional  Resources  (capacity) 

How  much  of  your  attentional  resources  were  you  supplying  to  the  session?  low _  high 

Were  you  making  the  greatest  possible  effort  (high)  or  giving  very  little 
attention  (low)? 

6.  Arousal 

How  aroused  were  you  in  the  session?  Were  you  alert  and  ready  for  activity  low _  high 

(high)  or  did  you  have  a  low  degree  of  alertness  (low)? 

7.  Concentration  of  Attention 

How  much  were  you  concentrating  on  the  session?  Were  you  bringing  all  low -  high 

your  thoughts  to  bear  (high)  or  was  your  attention  elsewhere  (low)? 

8.  Division  of  Attention 

How  much  was  your  attention  divided  in  the  session?  Were  you  low _  high 

concentrating  on  many  aspects  of  the  situation  (high)  or  focussed  on  only 
one  (low)? 

9.  Spare  Mental  Capacity 

How  much  mental  capacity  did  you  have  to  spare  in  this  session?  Did  you  low _  high 

have  sufficient  to  attend  to  many  new  variables  (high)  or  nothing  to  spare  at 
all  (low?) 

10.  Understanding  of  Session 

How  well  did  you  understand  the  session?  Did  you  understand  almost  low _  high 

everything  (high)  or  virtually  nothing  (low)? 

11.  Information  Quantity 

How  much  information  did  you  gain  tfom  your  environment  (inside  and  low _  high 

outside  the  cockpit)?  Did  you  receive  and  understand  a  great  deal  of 
knowledge  (high)  or  very  little  (low)? 

12.  Information  Quality 

How  good  was  the  information  you  had  gained  from  your  environment  low -  high 

(inside  and  outside  the  cockpit)?  Was  the  knowledge  communicated  very 
useful  (high)  or  was  it  of  very  little  use  (low)? 

13.  Familiarity  with  Session 

How  familiar  were  you  with  the  session?  Did  you  have  a  great  deal  of  low -  high 

relevant  experience  (high)  or  was  it  a  new  session  (low)? 

14.  Confidence  in  Ratings 

How  confident  are  you  of  the  ratings  you  have  just  made?  Are  you  very  low _  high 

confident  (high),  or  not  very  confident  (low)? 

Figure  18:  Situation  Awareness  Rating  Technique  Input  Form 
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F.3.2  Workload 


It  is  proposed  to  measures  workload  using  a  modified  version  of  the  NASA  Task  Load  Index 
(TLX)  (see  Figure  19).  In  addition  to  the  six  workload  subscales  used  in  the  standard  NASA 
TLX,  the  version  of  the  TLX  shown  here  also  includes  a  seventh  subscale  that  measures  the 
operator’s  overall  workload.  Specifically,  operators  would  be  asked  to  rate  their  workload  for: 

1 .  Mental  Demand 

2.  Physical  Demand 

3.  Temporal  Demand 

4.  Frustration  Level 

5.  Effort 

6.  Performance 

7.  Overall 

Operators  indicate  their  workload  by  placing  an  “X”  on  a  horizontal  scale  where  the  leftmost 
marker  represents  0%  (i.e.,  very  low)  workload  and  the  rightmost  marker  represents  100%  (i.e., 
very  high)  workload.  Each  10%  interval  is  demarcated  on  the  scale.  If  an  “X”  is  placed  in 
between  two  10%  demarcations  (e.g.,  between  50%  and  60%),  then  workload  will  be  measured  to 
the  nearest  5%  (i.e.,  55%).  Each  of  the  seven  workload  subscales  listed  above  will  be  rated 
separately.  That  is,  operators  will  place  an  “X”  on  the  Mental  Demand  scale  (where  0% 
represents  no  mental  demand  and  100%  represents  very  high  mental  demand).  Then,  they  will 
place  an  “X’  on  the  Physical  Demand  scale.  This  process  will  continue  until  they  have  placed  an 
“X”  on  all  seven  subscales. 
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Instructions:  Please  respond  to  each  statement  below  by  selecting  the  most  appropriate  rating  response 

based  on  your  experiences  today. 


1.  Mental  Demand:  How  much  mental  and  perceptual  activity  was  required  (e.g.,  thinking,  deciding, 
calculating,  remembering,  looking,  searching)?  Was  the  task  easy  or  demanding,  simple  or  complex, 
exacting  or  forgiving? 

LOW  I _ I _ I _ I _ I _ I _ I _ I _ I _ I _ IHIGH 

0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 

2.  Temporal  Demand:  How  much  time  pressure  did  you  feel  due  to  the  rate  or  pace  at  which  the  tasks  or 
task  elements  occurred?  Was  the  pace  slow  and  leisurely  or  rapid  and  frantic? 

LOW  I _ I _ I _ I _ I _ I _ I _ I _ I _ I _ IHIGH 

0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 

3.  Effort:  How  hard  did  you  have  to  work  (mentally  and  physically)  to  accomplish  your  level  of 
performance? 

LOW  I _ I _ I _ I _ I _ I _ I _ I _ I _ I _ IHIGH 

0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 

4.  Frustration  Level:  How  insecure,  discouraged,  irritated,  stressed,  and  annoyed  versus  secure, 
gratified,  content,  relaxed  and  complacent  did  you  feel  during  the  task? 

LOW  I _ I _ I _ I _ I _ I _ I _ I _ I _ I _ IHIGH 

0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 

5.  Performance:  How  successful  do  you  think  you  were  in  accomplishing  the  goals  of  the  task  set  by  the 
experimenter  (or  yourself)?  How  satisfied  were  you  with  your  performance  in  accomplishing  these 
goals? 

LOW  I _ I _ I _ I _ I _ I _ I _ I _ I _ I _ IHIGH 

0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 

6.  Overall  Workload:  Overall,  how  demanding  did  you  find  the  task? 

LOW  I _ I _ I _ I _ I _ I _ I _ I _ I _ I _ IHIGH 

0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 

Figure  19:  Proposed  Task  Load  Index  (TLX) 

F.3.3  Task-Relevant  Performance 

Approaches  for  measurement  of  task-relevant  performance  include: 

1.  Critical  Task  Sequence  (CTS)  completion  time:  CTSs  must  be  defined  a  priori  by  SMEs 
and  the  experimental  team  and  will  represent  a  critical  mission  task  that  the  operator  must 
complete  quickly  and  accurately  in  order  to  ensure  ship  survivability.  These  CTSs  should  also 
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be  chosen  based  on  their  likelihood  of  being  differentially  affected  by  the  independent 
variables  or  experimental  conditions.  Further  these  CTSs  must  have  objective  and  easily 
observable  start  and  end  points  that  can  be  precisely  captured  by  either  the  audio/video 
equipment  or  the  scenario  generation  tools  (e.g.,  button  press  responses,  verbal  commands). 
The  time  difference  between  the  start  and  end  points  of  each  CTS  should  be  calculated  so  as 
to  determine  the  length  of  time  required  to  complete  the  tasks. 

2.  Percentage  of  CTS  shedding:  The  number  of  incomplete  CTSs  relative  to  the  total  number 
of  CTSs  can  also  be  measured  to  compliment  the  CTS  completion  time  data.  It  is  standard 
practice  to  measure  error  rates  when  completion  time  (reaction  time)  for  a  task  is  a  dependent 
variable.  The  rationale  for  doing  so  is  to  capture  the  fact  that  some  operators  will  modify  their 
response  strategy  under  high  workload  conditions  where  they  trade-off  accuracy  for  speed. 
That  is,  they  selectively  respond  to  only  a  few  tasks  very  effectively  (i.e.,  low  completion 
times  and  error  rates),  but  at  the  cost  of  making  errors  on  (or  ignoring)  other  tasks.  In  the 
context  of  this  experiment,  the  percentage  of  CTS  shedding  would  effectively  capture  the 
strategy  described  above.  That  is,  some  operators  may  adopt  a  strategy  where  they  only 
complete  a  select  few  of  the  CTSs  (albeit  effectively),  but  do  so  at  the  cost  of  shedding  many 
other  CTSs.  Operators  that  use  this  strategy  will  score  well  in  the  CTS  completion  time 
measure,  but  will  consequently  score  poorly  for  CTS  shedding. 

F.3.4  Trust 

In  the  context  of  complex,  human-machine  systems,  trust  is  defined  as:  “the  extent  to  which  a 
user  is  confident  in  and  willing  to  act  on  the  basis  of  the  recommendations,  actions,  and  decisions 
of  an  artificially  intelligent  decision  aid”  [7].  Flowever,  trust  is  not  a  simple  uni-dimensional 
variable.  It  is  possible  to  be  correctly  distrusting  of  a  system  (e.g.,  when  it  is  unreliable),  but  also 
to  be  too  trusting  (‘over-trusting’  or  complacent)  or  not  trusting  enough  (‘under-trusting’  or 
sceptical). 

Participant  trust  and  acceptance  of  automation  is  determined  from  the  outcome  of  a  comparison 
process  between  the  perceived  reliability  of  the  automated  aid  (i.e.,  trust  in  aid)  and  the  perceived 
reliability  of  manual  control  (i.e.,  trust  in  self).  Decision  making  quality  will  increase  when  the 
participant  is  able  to  compare  the  abilities  of  the  CDSC  with  their  own  abilities.  A  subjective 
assessment  based  on  one  of  the  most  recent  and  a  comprehensive  model  of  trust  (i.e.,  model  of 
Human-Computer  Trust  [2])  is  recommended  to  measure  the  degree  of  participant  trust  in  the 
CDSC.  Overall  trust  in  the  CDSC  is  determined  by  cognition-based  trust  (i.e.,  trust  relating  to  the 
participant’s  perception  of  the  automation)  and  affect-based  trust  (i.e.,  trust  relating  to  the 
participant’s  emotive  response  to  automation).  Three  factors  underpin  cognition-based  trust  (i.e., 
perceived  understandability,  technical  competence,  and  reliability  [of  the  system]),  and  two 
factors  underpin  affect-based  trust  (i.e.,  faith  [in  the  system]  and  personal  attachment  [to  the 
system]).  Each  of  these  five  factors  has  five  sub-items  as  shown  in  Figure  20 
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1.  Perceived  rstiability 

R1 .  "he  system  alv/ays  provides  the  advice  I  "ec uire  to  r-ake  my  dec  sion. 

R2.  The  system  peffor~h3  reliaoly. 

R2.  The  system  resoor  ds  the  sa-he  way  jncer  the  same  cor-didohs  at  different  t~ies. 

R4.  can  rely  on  the  system  to  f^ncticr  orope'ly. 

R=.  "he  system  analyzes  p'ob  ems  cans  stendy. 

2.  PerceiVed  technical  competence 

T1.  "he  system  uses  opp-aphate  ■hetnocs  x  reach  cecisions 

T2.  The  system  has  sound  khcv/edge  aaott  this  type  cf  p'abletn  b.ilt  into  it. 

T3.  "he  advice  the  system  p'Pduces  is  as  good  as  that  ■.vhich  a  high  y  ccmpex'-t 
person  conic  produce. 

T4.  The  system  correctly  tses  the  information  I  enter. 

T£.  The  system  ma'tes  .se  o' all  the  Knowledge  and  information  ava  able  to  it  to 
produce  its  so  uticn  to  the  problem. 

3.  Perceived  undersxandablltty 

U1.  I  know  v-hatwil  happen  the  next  time  I  v.6e  the  system  Oecause  i  nnde'stohd 
hew  1  behaves. 

U2.  I  understand  how  the  syste-n  wi  assist  me  with  cecisions  I  have  to  make. 

U2.  Although  I  may  not  khov/  exaoty  how  the  system  \var<s  I  know  hov/  to  use  t  to 
make  cecisions  about  the  problem. 

U4.  It  is  easy  to  follow  what  the  system  dees. 

Us.  I  recognize  what  I  shoulc  co  to  get  the  advice  need  from  the  system  the  next 
t  me  I  use  it. 

4.  Faith 

FI .  I  believe  advee  f'om  the  system  even  when  I  don't  know  for  certain  that  it  is 
comect. 

F2.  When  I  am  uncertain  about  a  decision  I  believe  the  system  rat-e-  than  myself 

F3.  If  I  am  not  sure  about  a  dec  sion,  i  nave  'a  t”  that  the  system  v/ill  orevde  the  oest 
solut  on. 

F4.  When  the  system  gives  unusja  advice  am  confident  that  the  advice  is  co'rect. 

F5.  Even  if  I  have  hc  reason  to  expect  the  system  will  be  able  to  solve  a  ciffiCv.lt 
problem.  I  st  II  feel  ceiain  that  it  will 

5.  Personal  attachment 

P"  I  woulc  'eel  a  sense  of  oss  f  the  system  was  uhovailaole  and  could  nc  longer 
use  It 

P2.  I  'eel  a  sense  of  anaenment  to  us  ng  the  system. 

Fo.  I  find  the  system  Svitab  e  to  my  style  cf  decisiof-max  ng. 

P4  1 1  <e  using  the  system  for  decis  on-r-aking 

P5  I  have  a  personal  pre'e'ence  for  making  decisions  'with  the  system. 

Figure  20.  Human-Computer  Trust  rating  scale  [7] 

F.4  Experimental  Plan 

F.4.1  General 

The  following  sections  provide  an  overview  of  the  proposed  experimental  approach  to  evaluate 
the  Threat  Evaluation  (TE)  and  Combat  Power  Management  (CPM)  functional  displays. 
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F.4.2  Experimental  Design 

A  2  (Scenario  Complexity:  low  vs.  high)  x  2  (Decision  Support  System  [DSS]:  with  vs.  without) 
within-subjects  design  is  proposed. 

1.  Scenario  Complexity:  Complexity  refers  to  the  type,  number,  tactics,  location  and  timing  of 
threats  within  the  experimental  scenario.  The  high  complexity  scenario  should  possess: 

a.  An  increased  number  of  medium  and  high  level  threats  requiring  continual 
assessment  and  prioritization;  and 

b.  An  increased  number  of  targets  requiring  the  application  of  combat  power  resources. 

c.  Reduction  in  time  to  plan  and  apply  combat  power  in  response  to  an  incoming  target. 

2.  With  and  Without  Decision  Support:  Refers  to  the  presence  of  the  functional  displays  to 
support  the  CPM  decision  making  process.  Specifically,  the  ‘without  DSS’  condition  would 
only  entail  a  physical  display  representing  the  position  of  the  threats  (and  friendly  contacts). 

This  2x2  design  yields  four  experimental  conditions.  Given  that  both  Decision  Support  and 
Scenario  Complexity  would  be  within-subjects  factors,  each  operator  should  be  tested  under  all 
levels  of  those  factors. 

In  the  high-complexity  scenario,  operators  monitor  and  assess  a  larger  number  of  threats  as 
compared  to  the  low-complexity  scenario.  Further,  in  the  low-complexity  scenario,  operators 
engage  one  target  at  a  time  and  in  the  high-complexity  scenario  operators  engage  multiple  targets. 
The  increased  complexity  in  the  scenario  is  intended  to  introduce  additional  demands  on  the 
operator  with  respect  to  the  conduct  of  threat  evaluation  and  application  of  combat  power 
activities. 

When  operators  are  using  the  decision  support  system,  they  would  use  the  TE  and  CPM 
functional  views  as  well  as  the  physical  display  to  monitor  contacts,  threats,  and  targets.  When 
operators  are  not  using  the  decision  support  system,  they  would  be  limited  to  the  physical  display 
to  monitor  contacts,  threats,  and  targets. 

F.4.3  Experimental  Hypotheses 

Experimental  hypotheses,  in  the  form  of  predicted  outcomes,  are  presented  in  relation  to  the 
independent  variables  and  measures  of  performance  discussed  above  for  both  the  TE  and  CPM 
functional  displays.  The  precise  definition  of  critical  task  sequences  depends  on  the  scenarios  and 
the  capabilities  of  the  prototype.  Two  generic  tasks  that  require  minimal  interaction  with  the 
system  and  minimal  programming  of  the  INCOMMANDS  prototype  are  proposed  in  this 
experimental  plan  for  TE  and  for  CPM.  The  TE  tasks  are  to  query  the  operators  at  pre-set 
intervals  during  the  scenario  to  record  the  top  three  threats  as  well  as  having  them  identify  pre¬ 
defined  events  (e.g.,  threat  category  jumps).  The  CPM  tasks  are  to  report  which  of  the  ship’s 
combat  power  resources  to  employ  to  neutralize  a  target  and  to  recommend  when  to  launch  a 
combat  power  resource  to  neutralize  a  target.  Both  accuracy  and  response  time  for  these  tasks 
should  be  measured. 
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F.4.3.1  Threat  Evaluation 


Scenario  Complexity 

With  respect  to  the  TE  functional  display,  increasing  scenario  complexity  will: 

1.  Decrease  the  operator’s  SA  due  to  information  overload.  Operator  SA  should  be  assessed 
using  the  SART.  Each  operator  would  be  asked  to  rate  their  SA  for  critical  entities/events  that 
were  present  or  occurred  in  the  most  recently  completed  experimental  scenario. 

2.  Increase  the  operator’s  workload  due  to  more  information  being  processed.  Operator 
workload  should  be  measured  with  the  modified  form  of  the  NASA  TEX  discussed  above.  It 
will  assess  whether  or  not  the  scenario  complexity  manipulation  is  having  its  intended  effect 
(i.e.,  workload  ratings  should  increase  as  scenario  complexity  increases).  The  performance 
measures  would  need  to  be  used  to  and  the  usability  and  utility  questionnaires  were  used  to 
assess  it  will  highlight  any  OMl  deficiencies.  For  example,  if  certain  aspects  of  the  OMl  are 
difficult  to  use  or  interpret  under  certain  conditions,  then  this  will  be  reflected  in  the 
operator’s  rating  of  their  frustration  level. 

3.  Increase  the  time  required  by  the  operator  to  initiate  and/or  complete  task-relevant 
performance  due  to  information  overload.  Thus  time  related  task-relevant  performance  would 
the  operator’s  delays  in  reporting  what  their  top  three  threats  are  and  when  a  contact  is 
promoted  from  a  mid-level  threat  to  a  high-level  threat. 

4.  Decrease  the  accuracy  of  the  operator’s  task-relevant  performance  due  to  information 
overload.  Task-relevant  performance  would  be  the  number  of  errors  operators  make  when 
reporting  what  their  top  three  threats  are  and  when  a  contact  moves  from  a  mid-level  threat  to 
a  high-level  threat. 

Decision  Support  (with  vs.  without) 

The  presence  of  the  TE  functional  display  which  includes,  but  is  not  limited  to,  a  visual 

representation  of  prioritized  and  categorized  threats  as  well  as  temporal  information  regarding  the 

imminence  of  each  threat  should: 

1.  Increase  the  operator’s  SA  since  the  system  will  assist  the  operator  with  anticipating  such 
events  as  the  emergence  of  threats  as  they  migrate  across  threat  categorization.  Each  operator 
should  be  asked  to  rate  their  SA  (using  the  SART)  for  critical  entities/events  that  were  present 
or  occurred  in  the  most  recently  completed  experimental  scenario.  Further,  the  provision  for 
operator  anticipation  afforded  by  the  threat  visual  representation  will  allow  them  to  use 
‘down  time’  (e.g.,  during  a  lull  in  the  scenario)  to  plan  for  potential  future  events  (such  as  the 
requirement  to  report  the  top  three  threats).  Thus,  if  these  potential  events  do  occur 
(especially  if  they  occur  when  unforeseen  events  present  themselves),  then  the  anticipatory 
planning  will  serve  to  allow  the  operators  to  maintain  a  higher  level  of  SA  under  these 
conditions. 

2.  Decrease  the  operator’s  workload  since  the  system  will  be  performing  activities  typically 
delegated  to  the  operator.  The  modified  NASA  TEX  should  be  used  to  assess  operator 
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workload.  It  provides  a  method  to  assess  whether  or  not  the  deeision  support  system  is 
having  its  intended  effect  (i.e.,  workload  ratings  should  decrease  with  the  presence  of  the  TE 
functional  display).  Further,  the  provision  for  anticipatory  planning  afforded  by  the  threat 
functional  display  will  allow  operators  to  plan  for  future  events.  If  and  when  these  ‘future’ 
events  occur,  operators  will  be  better  equipped  to  deal  with  them,  especially  if  these  events 
occur  during  a  complex  part  of  the  scenario. 

3.  Decrease  the  time  required  by  the  operator  to  initiate  and/or  complete  task-relevant 
performance  because  the  TE  functional  display  provides  operators  with  greater,  more 
detailed,  and  more  quickly  and  easily  interpretable  information  (e.g.,  temporal  information)  to 
identify  and  prioritize  threats.  In  addition,  the  visual  representation  of  temporal  information 
relating  to  each  entity’s  ‘threat  trajectory’  over  time  should  facilitate  an  operator’s  ability  to 
accurately  anticipate  if  (and  when)  a  medium  level  threat  will  become  a  high  level  threat  (i.e., 
a  ‘threat  category  jump’). 

4.  Increase  the  accuracy  of  the  operator’s  task-relevant  performance  because  the  TE  functional 
display  provides  operators  with  greater,  more  detailed  and  more  quickly  and  easily 
interpretable  information  (e.g.,  temporal  information)  to  identify  and  prioritize  threats. 

F.4.3.2  Combat  Power  Management 

It  is  anticipated  that  most  (if  not  all)  of  the  direct  measures  identified  as  part  of  the  threat 

evaluation  hypotheses  will  apply  to  the  application  of  combat  power  function. 

Scenario  Complexity 

With  respect  to  the  CPM  functional  view,  increasing  scenario  complexity  will: 

1.  Decrease  the  operator’s  SA  due  to  information  overload.  This  will  make  it  more  difficult  for 
the  operator  to  continually  monitor  the  system-generated  engagement  plans.  Operator  SA 
should  be  assessed  using  the  SART. 

2.  Increase  the  operator’s  workload  due  to  the  requirement  for  additional  information  to  be 
processed.  Operator  workload  should  be  measured  with  the  modified  NASA  TEX.  It  will 
assess  whether  or  not  the  scenario  complexity  manipulation  is  having  its  intended  effect  (i.e., 
workload  ratings  should  increase  as  scenario  complexity  increases).  Second,  they  will 
highlight  any  OMl  deficiencies.  For  example,  if  certain  aspects  of  the  OMl  are  difficult  to  use 
or  interpret  under  certain  conditions,  then  this  will  be  reflected  in  the  operator’s  rating  of  their 
frustration  level. 

3.  Increase  the  time  required  by  the  operator  to  initiate  and/or  complete  task-relevant 
performance  due  to  information  overload.  Time  related  task-relevant  performance  should  be 
assessed  by  measuring  operator  delays  when  reporting  which  of  the  ship’s  combat  power 
resources  to  employ  to  neutralize  a  target  and  when  to  launch  a  combat  power  resource  to 
neutralize  a  target. 

4.  Decrease  the  accuracy  of  the  operator’s  task-relevant  performance  due  to  information 
overload.  Task-relevant  performance  should  be  assessed  by  measuring  the  number  of  errors 
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operators  make  when  reporting  which  of  the  ship’s  combat  power  resources  to  employ  to 
neutralize  a  threat. 

Decision  Support  (with  vs.  without) 

The  presence  of  the  CPM  functional  display  which  includes,  but  is  not  limited  to,  a  visual 
representation  of  engagement  plans  as  well  as  temporal  information  regarding  the  execution  of 
atomic  actions  should: 

1.  Increase  the  operator’s  SA  since  they  will  be  more  visually  aware  of  engagement  plans  and 
their  execution.  Each  operator  should  be  asked  to  rate  their  SA  (using  the  SART)  for  critical 
entities/events  that  were  present  or  occurred  in  the  most  recently  completed  experimental 
scenario.  Further,  the  provision  for  operator  anticipation  afforded  by  the  CPM  functional 
view  will  allow  them  to  use  ‘down  time’  (e.g.,  during  a  lull  in  the  scenario)  to  plan  for 
potential  future  events.  Thus,  if  these  potential  events  do  occur  (especially  if  they  occur  when 
unforeseen  events  present  themselves),  then  the  anticipatory  planning  will  serve  to  allow  the 
operators  to  maintain  a  higher  level  of  SA  under  these  conditions. 

2.  Decrease  the  operator’s  workload.  The  modified  NASA  TLX  should  be  used  to  assess 
operator  workload.  It  will  assess  whether  or  not  the  CPM  functional  display  is  having  its 
intended  effect  (i.e.,  workload  ratings  should  decrease  with  the  CPM  functional  display). 

3.  Decrease  the  time  required  by  the  operator  to  initiate  and/or  complete  task-relevant 
performance  because  the  CPM  functional  display  will  provide  operators  with  greater,  more 
detailed,  and  more  quickly  and  easily  interpretable  information  (e.g.,  temporal  information)  to 
decide  which  of  the  ship’s  combat  power  resources  to  employ  to  neutralize  a  threat.  Increase 
the  accuracy  of  the  operator’s  task-relevant  performance  because  the  CPM  functional  display 
will  provide  operators  with  greater,  more  detailed,  and  more  quickly  and  easily  interpretable 
information  (e.g.,  temporal  information)  to  identify  which  of  the  ship’s  combat  resources  to 
use  and  when  to  deploy  the  combat  resource  to  ensure  the  threat  is  neutralized. 

F.4.3.3  Scenario  Complexity  by  Decision  Support  Interaction 

It  is  assumed  that  the  interaction  between  Scenario  Complexity  and  Decision  Support  described 
below  will  not  be  differentially  mediated  by  TE  and  CPM  functional  displays.  As  such,  the 
anticipated  results  are  expected  to  take  the  same  form. 

1.  Operators  using  the  DSS  should  be  less  impaired  by  high  levels  of  scenario  complexity  than 
those  not  using  the  DSS  due  to  the  availability  of  temporal  information.  This  predicted 
interaction  is  depicted  in  Figure  21.  One  explanation  behind  this  claim  is  that  the  temporal 
information  provided  by  the  DSS  would  typically  be  maintained  in  the  operator’s  working 
memory  if  this  information  was  not  made  explicitly  available  to  them.  When  using  the  DSS, 
the  operator  does  not  have  to  maintain  this  information  in  working  memory,  which  therefore 
increases  memory  availability  for  other  tasks. 

2.  Operators  using  DSS  should  be  less  impaired  by  high  levels  of  scenario  complexity  than 
those  not  using  the  DSS  when  planning  and  applying  combat  power  resources.  Similar  to  the 
previous  item,  this  outcome  may  be  attributed  to  the  CPM  functional  display  representing 
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information  typically  retained  in  the  operator’s  working  memory.  In  the  low  scenario 
complexity  condition,  the  performance  difference  between  the  ‘with’  vs.  ‘without’  DSS 
conditions  will  not  be  as  large. 


Figure  21:  Anticipated  Interaction  Between  Independent  Variables 


F.4.4  Participants 

A  minimum  of  eight  active  and  qualified  CF  Naval  operators  between  1 8  and  60  years  would  be 
required  for  the  above  study.  Operators  should  be  either  qualified  as  Operations  Room  Operators 
(ORO)  or  Sensor  Weapons  Controllers  (SWC).  The  ORO  and  the  SWC  are  the  two  primary 
Combat  Power  Management  operators  (i.e.,  threat  evaluation,  engageability  assessment,  combat 
power  application)  within  the  operations  room  of  the  HALIFAX  Class  frigate.  The  ORO  is 
responsible  for  ensuring  that  the  appropriate  response  is  taken  to  counter  any  threat  encountered 
by  the  ship.  As  such,  the  ORO  evaluates  the  requirements  for  resources  of  each  warfare  area  (air, 
surface,  subsurface);  resolving  any  conflicts  that  arise,  to  most  effectively  allocate  those  resources 
when  conducting  multi-warfare  operations.  The  SWC  manages  two  warfare  areas  (surface  and 
air)  including  the  assessment  of  threats,  prioritization  of  targets,  performance  prediction  of  the 
above-water  weapons  and  sensors  as  well  as  the  planning  and  application  of  combat  power. 

All  operators  should  have  experience  with  the  CCS-330  simulator.  It  is  expected  that 
participating  operators  will  be  recruited  from  Halifax,  N.S.  and  will  conduct  the  tasks  while  on 
duty;  however,  participation  is  voluntary. 
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F.4.5  Equipment  and  Facilities 


The  proposed  experiment  is  designed  to  be  earried  out  using  the  INCOMMANDS  deeision 
support  system  prototype.  Video  cameras  will  be  required  to  record  each  trial. 

The  performance  measures  would  need  to  be  used  to  and  the  usability  and  utility  questionnaires 
were  used  to  assess  Scenarios  for  stimulation  of  the  individual  trials  during  the  experiment  will  be 
developed  using  STAGE  Scenario.  The  scenarios  should  have  the  following  characteristics: 

1.  Be  complete,  coherent  and  allow  key  operator  performance  aspects  to  be  measured  (as 
defined  a  priori  by  the  experimental  team).  For  example,  if  the  operator’s  SA  is  to  be 
measured  by  asking  them  to  identify  which  of  two  “medium”  threats  is  most  likely  to  next 
become  a  “high”  threat  (i.e.,  a  “threat  category  jump”),  then  the  scenario  must  include  a 
situation  where  there  are  at  least  two  “medium”  threats  that  are  strong  “threat  category  jump” 
candidates. 

2.  Accommodate  the  independent  manipulation  of  TE  and  CPM  complexity  as  they  are  likely  to 
be  differentially  affected.  For  example,  the  presence  of  a  single  threat  is  likely  to  produce  low 
complexity  for  CPM  but  could  produce  high  complexity  for  TE  if  there  are  many  non-hostile 
tracks  in  addition  to  this  single  threat. 

3.  Allow  each  operator  will  be  tested  on  both  Decision  Support  conditions  (with  vs.  without) 
and  under  both  Scenario  Complexities  (low  and  high).  Each  individual  operator  will 
experience  both  levels  of  Scenario  Complexity  twice  (once  with  the  DSS  and  once  without 
the  DSS).  If  the  same  (low  or  high)  complexity  scenario  is  used  twice,  the  operators’ 
memory  from  their  first  experience  with  a  scenario  will  influence  their  performance  during 
their  second  exposure  to  that  (same)  scenario.  In  order  to  avoid  this  experimental  confound,  a 
second  version  of  both  the  low  and  the  high  complexity  scenarios  must  be  created  by  taking 
the  original  scenario  and  rotating  its  mirror  image.  Track  identification  numbers  for  each 
contact  should  also  be  changed  in  order  to  further  increase  the  disparity  between  the  two 
scenario  versions. 

4.  The  two  versions  of  both  scenarios  should  be  sufficiently  different  that  any  changes  in 
operators’  performance  between  the  original  scenario  version  compared  to  the  rotated  mirror 
image  (i.e.,  modified)  version  cannot  be  attributed  to  the  operators’  memory  for  the  ground 
truth  from  their  first  experience  with  that  scenario.  As  such,  any  observable  difference 
between  the  original  and  modified  versions  of  the  scenario  can  be  more  confidently  attributed 
to  differences  between  DSS  conditions  (with  and  without).  Furthermore,  these  two  scenario 
versions  should  be  similar  enough  (e.g.,  same  number  and  relative  placement  of  unknown 
contacts)  that  both  versions  will  induce  similar  workloads  and  can  therefore  be  directly 
compared. 

F.4.6  Counterbalancing 

As  discussed  above,  operators  are  to  participate  in  all  four  experimental  conditions.  Since  the 
operators  will  not  have  extensive  experience  with  the  INCOMMANDS  system,  it  recommended 
that  scenario  complexity  not  be  counterbalanced.  That  is,  operators  should  always  receive  the  low 
complexity  scenarios  first,  once  using  the  no  DSS  and  once  using  the  DSS.  The  high  complexity 
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scenarios  will  then  always  occur  after  the  low  complexity  scenarios.  However,  DSS 
presence/absence  condition  should  be  counterbalanced  in  that  half  of  the  operators  receive  the 
without  DSS  condition  first  followed  by  the  with  DSS  condition.  The  order  of  the  DSS 
presence/absence  condition  would  be  reversed  for  the  other  half  of  the  operators.  Additionally, 
scenario  version  should  be  counterbalanced  across  operators.  Even  if  every  attempt  is  made  to 
equate  the  original  version  of  a  scenario  with  its  modified  version,  there  remains  the  possibility 
that  they  will  not  be  perfectly  matched.  Thus,  half  of  the  operators  should  receive  the  original 
scenario  version  first  and  the  modified  version  second.  This  order  should  be  reversed  for  the  other 
half  of  the  operators.  A  minimum  of  eight  operators  are  therefore  required  for  a  complete 
experimental  counterbalance.  The  details  of  this  counterbalancing  scheme  are  shown  in  Table  1. 

Table  1:  Counterbalancing  Details 


Counter¬ 

balance 

Experimental  Trial 

1 

2 

3 

4 

1 

Scenario 

Complexity 

LOW 

LOW 

HIGH 

HIGH 

OMl  Type 

NO  DSS 

DSS 

NO  DSS 

DSS 

Scenario  Version 

1 

2 

1 

2 

2 

Scenario 

Complexity 

LOW 

LOW 

HIGH 

HIGH 

OMl  Type 

NO  DSS 

DSS 

NO  DSS 

DSS 

Scenario  Version 

2 

1 

2 

1 

3 

Scenario 

Complexity 

LOW 

LOW 

HIGH 

HIGH 

OMl  Type 

DSS 

NO  DSS 

DSS 

NO  DSS 

Scenario  Version 

1 

2 

1 

2 

4 

Scenario 

Complexity 

LOW 

LOW 

HIGH 

HIGH 

OMl  Type 

DSS 

NO  DSS 

DSS 

NO  DSS 

Scenario  Version 

2 

1 

2 

1 
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List  of  symbols/abbreviations/acronyms/initialisms 


AWW 

Above  Water  Warfare 

C2 

Command  and  Control 

CCS 

Command  and  Control  System 

CDS 

Command  Decision  Support 

CDSC 

Command  Decision  Support  Concept 

CF 

Canadian  Forces 

CFMWC 

Canadian  Forces  Maritime  Warfare  Center 

CO 

Commanding  Officer 

COMDAT 

Command  Decision  Aiding  Technology 

CONORS 

Concept  of  Operations 

CP 

Combat  Power 

CPM 

Combat  Power  Management 

CTS 

Critical  Task  Sequence 

DRDC 

Defence  Research  and  Development  Canada 

DSS 

Decision  Support  System 

EO 

Electro-Optical 

ESM 

Electronic  Support  Measures 

ESS 

Electronic  Support  System 

HCl 

Fluman  Computer  Interaction 

HCM 

Flalifax  Class  Modernization 

HF 

Human  Factors 

HREC 

Human  Research  Ethics  Committee 

HSl 

Human  Systems  Integration 

INCOMMANDS 

Innovative  Naval  Combat  Management  Decision  Support 

MOE 

Measures  of  Effectiveness 

MOP 

Measures  of  Performance 

OMl 

Operator  Machine  Interface 

ORO 

Operations  Room  Officer 

PS 

Professional  Services 
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ROE 

RMP 

SA 

SAM 

SME 

SOP 

SOW 

SSM 

swc 

TAM 

TDP 

TE 

TEX 


Rules  of  Engagement 
Recognized  Maritime  Picture 
Situational  Awareness 
Surface-to-Air  Missile 
Subject  Matter  Expert 
Standard  Operating  Procedure 
Statement  of  Work 
Surface-to-Surface  Missile 
Sensor  Weapons  Controller 
Technology  Acceptance  Model 
Technology  Demonstration  Project 
Threat  Evaluation 
Task  Eoad  Index 
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